2023-03-24 01:53:31,675 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77 2023-03-24 01:53:31,690 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-24 01:53:31,706 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-24 01:53:31,707 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1, deleteOnExit=true 2023-03-24 01:53:31,707 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-24 01:53:31,708 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/test.cache.data in system properties and HBase conf 2023-03-24 01:53:31,708 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/hadoop.tmp.dir in system properties and HBase conf 2023-03-24 01:53:31,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/hadoop.log.dir in system properties and HBase conf 2023-03-24 01:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-24 01:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-24 01:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-24 01:53:31,827 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-24 01:53:32,251 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-24 01:53:32,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-24 01:53:32,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-24 01:53:32,256 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-24 01:53:32,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 01:53:32,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-24 01:53:32,257 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-24 01:53:32,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 01:53:32,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 01:53:32,258 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-24 01:53:32,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/nfs.dump.dir in system properties and HBase conf 2023-03-24 01:53:32,259 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/java.io.tmpdir in system properties and HBase conf 2023-03-24 01:53:32,260 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 01:53:32,260 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-24 01:53:32,260 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-24 01:53:32,852 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 01:53:32,857 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 01:53:33,171 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-24 01:53:33,347 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-24 01:53:33,364 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 01:53:33,403 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 01:53:33,439 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/java.io.tmpdir/Jetty_localhost_38887_hdfs____johtng/webapp 2023-03-24 01:53:33,570 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38887 2023-03-24 01:53:33,609 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 01:53:33,610 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 01:53:34,054 WARN [Listener at localhost/39055] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 01:53:34,197 WARN [Listener at localhost/39055] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-24 01:53:34,231 WARN [Listener at localhost/39055] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 01:53:34,239 INFO [Listener at localhost/39055] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 01:53:34,246 INFO [Listener at localhost/39055] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/java.io.tmpdir/Jetty_localhost_34043_datanode____.jzgmln/webapp 2023-03-24 01:53:34,361 INFO [Listener at localhost/39055] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34043 2023-03-24 01:53:34,765 WARN [Listener at localhost/44783] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 01:53:35,321 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2e7739fd0e3b64be: Processing first storage report for DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871 from datanode 72b7d64b-790b-4d54-8523-2c6568c41f1c 2023-03-24 01:53:35,323 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2e7739fd0e3b64be: from storage DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871 node DatanodeRegistration(127.0.0.1:32871, datanodeUuid=72b7d64b-790b-4d54-8523-2c6568c41f1c, infoPort=45311, infoSecurePort=0, ipcPort=44783, storageInfo=lv=-57;cid=testClusterID;nsid=218677478;c=1679622812935), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-24 01:53:35,323 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2e7739fd0e3b64be: Processing first storage report for DS-5ff34951-1669-4394-9c00-c5a793dd21b5 from datanode 72b7d64b-790b-4d54-8523-2c6568c41f1c 2023-03-24 01:53:35,324 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2e7739fd0e3b64be: from storage DS-5ff34951-1669-4394-9c00-c5a793dd21b5 node DatanodeRegistration(127.0.0.1:32871, datanodeUuid=72b7d64b-790b-4d54-8523-2c6568c41f1c, infoPort=45311, infoSecurePort=0, ipcPort=44783, storageInfo=lv=-57;cid=testClusterID;nsid=218677478;c=1679622812935), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-24 01:53:35,373 DEBUG [Listener at localhost/44783] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77 2023-03-24 01:53:35,440 INFO [Listener at localhost/44783] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1/zookeeper_0, clientPort=62686, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-24 01:53:35,453 INFO [Listener at localhost/44783] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=62686 2023-03-24 01:53:35,461 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:35,463 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:36,142 INFO [Listener at localhost/44783] util.FSUtils(471): Created version file at hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97 with version=8 2023-03-24 01:53:36,142 INFO [Listener at localhost/44783] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/hbase-staging 2023-03-24 01:53:36,465 INFO [Listener at localhost/44783] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-24 01:53:36,949 INFO [Listener at localhost/44783] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-24 01:53:36,981 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:36,982 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:36,982 INFO [Listener at localhost/44783] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 01:53:36,982 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:36,982 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 01:53:37,124 INFO [Listener at localhost/44783] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 01:53:37,199 DEBUG [Listener at localhost/44783] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-24 01:53:37,291 INFO [Listener at localhost/44783] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35345 2023-03-24 01:53:37,301 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:37,304 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:37,328 INFO [Listener at localhost/44783] zookeeper.RecoverableZooKeeper(93): Process identifier=master:35345 connecting to ZooKeeper ensemble=127.0.0.1:62686 2023-03-24 01:53:37,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:353450x0, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 01:53:37,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:35345-0x101143a28860000 connected 2023-03-24 01:53:37,399 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 01:53:37,400 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 01:53:37,403 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 01:53:37,411 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35345 2023-03-24 01:53:37,412 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35345 2023-03-24 01:53:37,412 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35345 2023-03-24 01:53:37,413 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35345 2023-03-24 01:53:37,413 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35345 2023-03-24 01:53:37,420 INFO [Listener at localhost/44783] master.HMaster(444): hbase.rootdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97, hbase.cluster.distributed=false 2023-03-24 01:53:37,495 INFO [Listener at localhost/44783] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-24 01:53:37,495 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:37,495 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:37,495 INFO [Listener at localhost/44783] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 01:53:37,495 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 01:53:37,496 INFO [Listener at localhost/44783] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 01:53:37,500 INFO [Listener at localhost/44783] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 01:53:37,503 INFO [Listener at localhost/44783] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44439 2023-03-24 01:53:37,505 INFO [Listener at localhost/44783] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-24 01:53:37,512 DEBUG [Listener at localhost/44783] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-24 01:53:37,513 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:37,516 INFO [Listener at localhost/44783] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:37,517 INFO [Listener at localhost/44783] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44439 connecting to ZooKeeper ensemble=127.0.0.1:62686 2023-03-24 01:53:37,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:444390x0, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 01:53:37,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44439-0x101143a28860001 connected 2023-03-24 01:53:37,522 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 01:53:37,523 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 01:53:37,524 DEBUG [Listener at localhost/44783] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 01:53:37,525 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44439 2023-03-24 01:53:37,525 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44439 2023-03-24 01:53:37,526 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44439 2023-03-24 01:53:37,526 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44439 2023-03-24 01:53:37,527 DEBUG [Listener at localhost/44783] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44439 2023-03-24 01:53:37,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:37,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 01:53:37,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:37,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 01:53:37,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 01:53:37,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:37,569 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 01:53:37,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,35345,1679622816283 from backup master directory 2023-03-24 01:53:37,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 01:53:37,574 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:37,574 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 01:53:37,575 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 01:53:37,575 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:37,578 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-24 01:53:37,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-24 01:53:38,072 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/hbase.id with ID: 2ab9d07a-e473-4c8d-9b23-58e66cd59c22 2023-03-24 01:53:38,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 01:53:38,126 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:38,566 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x26fc59c7 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:38,600 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f983beb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:38,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:53:38,631 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-24 01:53:38,650 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-24 01:53:38,650 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-24 01:53:38,652 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:38,656 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:38,657 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 01:53:38,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store-tmp 2023-03-24 01:53:38,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:38,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 01:53:38,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:53:38,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:53:38,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 01:53:38,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:53:38,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:53:38,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 01:53:38,722 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/WALs/jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:38,744 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35345%2C1679622816283, suffix=, logDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/WALs/jenkins-hbase4.apache.org,35345,1679622816283, archiveDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/oldWALs, maxLogs=10 2023-03-24 01:53:38,811 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK] 2023-03-24 01:53:38,820 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:38,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/WALs/jenkins-hbase4.apache.org,35345,1679622816283/jenkins-hbase4.apache.org%2C35345%2C1679622816283.1679622818754 2023-03-24 01:53:38,895 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK]] 2023-03-24 01:53:38,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:53:38,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:38,900 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:38,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:38,983 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:38,991 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-24 01:53:39,022 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-24 01:53:39,038 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:39,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:39,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:39,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 01:53:39,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:53:39,068 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74159451, jitterRate=0.10506193339824677}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 01:53:39,068 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 01:53:39,069 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-24 01:53:39,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-24 01:53:39,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-24 01:53:39,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-24 01:53:39,096 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-24 01:53:39,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-03-24 01:53:39,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-24 01:53:39,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-24 01:53:39,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-24 01:53:39,199 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-24 01:53:39,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-24 01:53:39,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-24 01:53:39,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-24 01:53:39,219 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-24 01:53:39,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:39,224 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-24 01:53:39,225 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-24 01:53:39,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-24 01:53:39,241 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 01:53:39,241 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 01:53:39,241 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:39,242 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,35345,1679622816283, sessionid=0x101143a28860000, setting cluster-up flag (Was=false) 2023-03-24 01:53:39,256 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-24 01:53:39,264 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:39,269 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:39,274 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-24 01:53:39,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:39,278 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.hbase-snapshot/.tmp 2023-03-24 01:53:39,332 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(952): ClusterId : 2ab9d07a-e473-4c8d-9b23-58e66cd59c22 2023-03-24 01:53:39,338 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-24 01:53:39,345 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-24 01:53:39,346 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-24 01:53:39,349 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-24 01:53:39,350 DEBUG [RS:0;jenkins-hbase4:44439] zookeeper.ReadOnlyZKClient(139): Connect 0x64a6faa1 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:39,356 DEBUG [RS:0;jenkins-hbase4:44439] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a611845, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:39,357 DEBUG [RS:0;jenkins-hbase4:44439] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e5e91d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-24 01:53:39,382 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:44439 2023-03-24 01:53:39,388 INFO [RS:0;jenkins-hbase4:44439] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-24 01:53:39,388 INFO [RS:0;jenkins-hbase4:44439] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-24 01:53:39,388 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1023): About to register with Master. 2023-03-24 01:53:39,390 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,35345,1679622816283 with isa=jenkins-hbase4.apache.org/172.31.14.131:44439, startcode=1679622817494 2023-03-24 01:53:39,398 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-24 01:53:39,407 DEBUG [RS:0;jenkins-hbase4:44439] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-24 01:53:39,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-24 01:53:39,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679622849410 2023-03-24 01:53:39,412 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-24 01:53:39,416 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 01:53:39,417 INFO [PEWorker-2] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-24 01:53:39,422 INFO [PEWorker-2] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 01:53:39,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-24 01:53:39,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-24 01:53:39,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-24 01:53:39,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-24 01:53:39,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-24 01:53:39,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,433 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-24 01:53:39,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-24 01:53:39,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-24 01:53:39,437 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-24 01:53:39,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-24 01:53:39,439 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679622819439,5,FailOnTimeoutGroup] 2023-03-24 01:53:39,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679622819439,5,FailOnTimeoutGroup] 2023-03-24 01:53:39,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-24 01:53:39,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,457 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 01:53:39,458 INFO [PEWorker-2] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 01:53:39,459 INFO [PEWorker-2] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97 2023-03-24 01:53:39,468 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55769, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-24 01:53:39,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,496 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97 2023-03-24 01:53:39,496 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39055 2023-03-24 01:53:39,496 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-24 01:53:39,502 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 01:53:39,502 DEBUG [RS:0;jenkins-hbase4:44439] zookeeper.ZKUtil(162): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,502 WARN [RS:0;jenkins-hbase4:44439] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 01:53:39,503 INFO [RS:0;jenkins-hbase4:44439] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 01:53:39,503 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/WALs/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,504 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44439,1679622817494] 2023-03-24 01:53:39,514 DEBUG [RS:0;jenkins-hbase4:44439] zookeeper.ZKUtil(162): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,523 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-24 01:53:39,531 INFO [RS:0;jenkins-hbase4:44439] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-24 01:53:39,549 INFO [RS:0;jenkins-hbase4:44439] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-24 01:53:39,553 INFO [RS:0;jenkins-hbase4:44439] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-24 01:53:39,553 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,555 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-24 01:53:39,561 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,562 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-24 01:53:39,563 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,563 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,563 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,563 DEBUG [RS:0;jenkins-hbase4:44439] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-24 01:53:39,564 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,564 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,564 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,580 INFO [RS:0;jenkins-hbase4:44439] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-24 01:53:39,582 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44439,1679622817494-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:39,598 INFO [RS:0;jenkins-hbase4:44439] regionserver.Replication(203): jenkins-hbase4.apache.org,44439,1679622817494 started 2023-03-24 01:53:39,598 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44439,1679622817494, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44439, sessionid=0x101143a28860001 2023-03-24 01:53:39,598 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-24 01:53:39,599 DEBUG [RS:0;jenkins-hbase4:44439] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,599 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44439,1679622817494' 2023-03-24 01:53:39,599 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:39,599 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44439,1679622817494' 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-24 01:53:39,600 DEBUG [RS:0;jenkins-hbase4:44439] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-24 01:53:39,601 DEBUG [RS:0;jenkins-hbase4:44439] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-24 01:53:39,601 INFO [RS:0;jenkins-hbase4:44439] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-24 01:53:39,601 INFO [RS:0;jenkins-hbase4:44439] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-24 01:53:39,712 INFO [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44439%2C1679622817494, suffix=, logDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/WALs/jenkins-hbase4.apache.org,44439,1679622817494, archiveDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/oldWALs, maxLogs=32 2023-03-24 01:53:39,730 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK] 2023-03-24 01:53:39,735 INFO [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/WALs/jenkins-hbase4.apache.org,44439,1679622817494/jenkins-hbase4.apache.org%2C44439%2C1679622817494.1679622819714 2023-03-24 01:53:39,736 DEBUG [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK]] 2023-03-24 01:53:39,873 DEBUG [PEWorker-2] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:39,876 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 01:53:39,878 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/info 2023-03-24 01:53:39,879 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 01:53:39,880 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:39,880 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 01:53:39,883 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/rep_barrier 2023-03-24 01:53:39,884 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 01:53:39,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:39,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 01:53:39,888 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/table 2023-03-24 01:53:39,888 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 01:53:39,889 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:39,891 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740 2023-03-24 01:53:39,892 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740 2023-03-24 01:53:39,896 DEBUG [PEWorker-2] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:53:39,898 DEBUG [PEWorker-2] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 01:53:39,902 DEBUG [PEWorker-2] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:53:39,904 INFO [PEWorker-2] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69162905, jitterRate=0.030607596039772034}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:53:39,904 DEBUG [PEWorker-2] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 01:53:39,904 DEBUG [PEWorker-2] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 01:53:39,904 INFO [PEWorker-2] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 01:53:39,904 DEBUG [PEWorker-2] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 01:53:39,904 DEBUG [PEWorker-2] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 01:53:39,904 DEBUG [PEWorker-2] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 01:53:39,906 INFO [PEWorker-2] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 01:53:39,906 DEBUG [PEWorker-2] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 01:53:39,911 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 01:53:39,911 INFO [PEWorker-2] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-24 01:53:39,921 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-24 01:53:39,937 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-24 01:53:39,939 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-24 01:53:40,091 DEBUG [jenkins-hbase4:35345] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-24 01:53:40,094 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44439,1679622817494, state=OPENING 2023-03-24 01:53:40,102 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-24 01:53:40,104 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:40,105 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 01:53:40,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:53:40,293 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:40,295 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-24 01:53:40,298 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-24 01:53:40,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-24 01:53:40,310 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 01:53:40,314 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44439%2C1679622817494.meta, suffix=.meta, logDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/WALs/jenkins-hbase4.apache.org,44439,1679622817494, archiveDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/oldWALs, maxLogs=32 2023-03-24 01:53:40,332 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK] 2023-03-24 01:53:40,336 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/WALs/jenkins-hbase4.apache.org,44439,1679622817494/jenkins-hbase4.apache.org%2C44439%2C1679622817494.meta.1679622820316.meta 2023-03-24 01:53:40,337 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32871,DS-0a0cc283-3fdb-41d6-a585-b03a1bb97871,DISK]] 2023-03-24 01:53:40,337 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:53:40,338 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-24 01:53:40,356 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-24 01:53:40,361 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-24 01:53:40,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-24 01:53:40,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:40,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-24 01:53:40,367 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-24 01:53:40,371 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 01:53:40,373 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/info 2023-03-24 01:53:40,373 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/info 2023-03-24 01:53:40,374 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 01:53:40,375 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:40,375 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 01:53:40,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/rep_barrier 2023-03-24 01:53:40,377 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/rep_barrier 2023-03-24 01:53:40,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 01:53:40,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:40,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 01:53:40,380 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/table 2023-03-24 01:53:40,380 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/table 2023-03-24 01:53:40,380 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 01:53:40,381 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:40,383 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740 2023-03-24 01:53:40,385 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740 2023-03-24 01:53:40,388 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:53:40,391 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 01:53:40,392 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71317381, jitterRate=0.06271179020404816}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:53:40,392 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 01:53:40,404 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679622820285 2023-03-24 01:53:40,424 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-24 01:53:40,425 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-24 01:53:40,426 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44439,1679622817494, state=OPEN 2023-03-24 01:53:40,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-24 01:53:40,430 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 01:53:40,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-24 01:53:40,434 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44439,1679622817494 in 323 msec 2023-03-24 01:53:40,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-24 01:53:40,440 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 514 msec 2023-03-24 01:53:40,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1240 sec 2023-03-24 01:53:40,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679622820445, completionTime=-1 2023-03-24 01:53:40,445 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-24 01:53:40,446 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-24 01:53:40,509 DEBUG [hconnection-0x2b20a9e4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:40,512 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58242, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:40,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-24 01:53:40,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679622880530 2023-03-24 01:53:40,530 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679622940530 2023-03-24 01:53:40,531 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-03-24 01:53:40,558 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35345,1679622816283-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:40,559 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35345,1679622816283-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:40,559 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35345,1679622816283-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:40,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:35345, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:40,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-24 01:53:40,568 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-24 01:53:40,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-24 01:53:40,578 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 01:53:40,587 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-24 01:53:40,590 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:53:40,593 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:53:40,621 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/hbase/namespace/341827374aceecc4b2088bebf463e093 2023-03-24 01:53:40,624 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/hbase/namespace/341827374aceecc4b2088bebf463e093 empty. 2023-03-24 01:53:40,625 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/hbase/namespace/341827374aceecc4b2088bebf463e093 2023-03-24 01:53:40,625 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-24 01:53:41,051 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-24 01:53:41,054 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 341827374aceecc4b2088bebf463e093, NAME => 'hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 341827374aceecc4b2088bebf463e093, disabling compactions & flushes 2023-03-24 01:53:41,467 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. after waiting 0 ms 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,467 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,467 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 341827374aceecc4b2088bebf463e093: 2023-03-24 01:53:41,472 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:53:41,490 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679622821475"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622821475"}]},"ts":"1679622821475"} 2023-03-24 01:53:41,525 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:53:41,527 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:53:41,533 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622821528"}]},"ts":"1679622821528"} 2023-03-24 01:53:41,538 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-24 01:53:41,548 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=341827374aceecc4b2088bebf463e093, ASSIGN}] 2023-03-24 01:53:41,551 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=341827374aceecc4b2088bebf463e093, ASSIGN 2023-03-24 01:53:41,552 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=341827374aceecc4b2088bebf463e093, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:53:41,703 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=341827374aceecc4b2088bebf463e093, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:41,704 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679622821703"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622821703"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622821703"}]},"ts":"1679622821703"} 2023-03-24 01:53:41,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 341827374aceecc4b2088bebf463e093, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:53:41,872 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,873 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 341827374aceecc4b2088bebf463e093, NAME => 'hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:53:41,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:41,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,876 INFO [StoreOpener-341827374aceecc4b2088bebf463e093-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,878 DEBUG [StoreOpener-341827374aceecc4b2088bebf463e093-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/info 2023-03-24 01:53:41,879 DEBUG [StoreOpener-341827374aceecc4b2088bebf463e093-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/info 2023-03-24 01:53:41,879 INFO [StoreOpener-341827374aceecc4b2088bebf463e093-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 341827374aceecc4b2088bebf463e093 columnFamilyName info 2023-03-24 01:53:41,880 INFO [StoreOpener-341827374aceecc4b2088bebf463e093-1] regionserver.HStore(310): Store=341827374aceecc4b2088bebf463e093/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:41,882 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,883 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,888 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 341827374aceecc4b2088bebf463e093 2023-03-24 01:53:41,892 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:53:41,893 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 341827374aceecc4b2088bebf463e093; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60698852, jitterRate=-0.0955166220664978}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 01:53:41,893 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 341827374aceecc4b2088bebf463e093: 2023-03-24 01:53:41,895 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093., pid=6, masterSystemTime=1679622821865 2023-03-24 01:53:41,899 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,899 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:53:41,901 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=341827374aceecc4b2088bebf463e093, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:41,902 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679622821901"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622821901"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622821901"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622821901"}]},"ts":"1679622821901"} 2023-03-24 01:53:41,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-24 01:53:41,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 341827374aceecc4b2088bebf463e093, server=jenkins-hbase4.apache.org,44439,1679622817494 in 195 msec 2023-03-24 01:53:41,913 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-24 01:53:41,913 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=341827374aceecc4b2088bebf463e093, ASSIGN in 361 msec 2023-03-24 01:53:41,915 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:53:41,915 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622821915"}]},"ts":"1679622821915"} 2023-03-24 01:53:41,918 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-24 01:53:41,923 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:53:41,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3440 sec 2023-03-24 01:53:41,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-24 01:53:41,995 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-24 01:53:41,995 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:53:42,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-24 01:53:42,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 01:53:42,056 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-03-24 01:53:42,067 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-24 01:53:42,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 01:53:42,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-24 01:53:42,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-24 01:53:42,095 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-24 01:53:42,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.520sec 2023-03-24 01:53:42,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-24 01:53:42,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-24 01:53:42,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-24 01:53:42,100 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35345,1679622816283-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-24 01:53:42,101 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35345,1679622816283-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-24 01:53:42,112 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-24 01:53:42,138 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:42,143 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:42,156 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:42,169 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58250, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:42,178 INFO [Listener at localhost/44783] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:53:42,201 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=256, ProcessCount=169, AvailableMemoryMB=6462 2023-03-24 01:53:42,217 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:53:42,220 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:53:42,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:53:42,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:53:42,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:53:42,237 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:53:42,239 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:53:42,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-24 01:53:42,242 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:42,243 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf empty. 2023-03-24 01:53:42,245 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:42,245 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:53:42,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 01:53:42,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 01:53:42,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 01:53:42,663 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:53:42,665 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 63c1038f18949b69fb052a65a9d44cbf, NAME => 'TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:53:42,677 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:42,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 63c1038f18949b69fb052a65a9d44cbf, disabling compactions & flushes 2023-03-24 01:53:42,678 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:42,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:42,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. after waiting 0 ms 2023-03-24 01:53:42,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:42,678 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:42,678 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:42,682 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:53:42,683 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622822683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622822683"}]},"ts":"1679622822683"} 2023-03-24 01:53:42,686 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:53:42,688 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:53:42,688 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622822688"}]},"ts":"1679622822688"} 2023-03-24 01:53:42,690 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:53:42,796 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, ASSIGN}] 2023-03-24 01:53:42,799 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, ASSIGN 2023-03-24 01:53:42,801 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:53:42,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 01:53:42,952 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=63c1038f18949b69fb052a65a9d44cbf, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:42,952 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622822952"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622822952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622822952"}]},"ts":"1679622822952"} 2023-03-24 01:53:42,956 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:53:43,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:43,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 63c1038f18949b69fb052a65a9d44cbf, NAME => 'TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:53:43,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:53:43,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,119 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,122 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A 2023-03-24 01:53:43,122 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A 2023-03-24 01:53:43,124 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:53:43,124 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63c1038f18949b69fb052a65a9d44cbf columnFamilyName A 2023-03-24 01:53:43,125 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(310): Store=63c1038f18949b69fb052a65a9d44cbf/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:43,125 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,128 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B 2023-03-24 01:53:43,128 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B 2023-03-24 01:53:43,128 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:53:43,128 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63c1038f18949b69fb052a65a9d44cbf columnFamilyName B 2023-03-24 01:53:43,129 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(310): Store=63c1038f18949b69fb052a65a9d44cbf/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:43,129 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,131 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C 2023-03-24 01:53:43,131 DEBUG [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C 2023-03-24 01:53:43,132 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:53:43,132 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 63c1038f18949b69fb052a65a9d44cbf columnFamilyName C 2023-03-24 01:53:43,133 INFO [StoreOpener-63c1038f18949b69fb052a65a9d44cbf-1] regionserver.HStore(310): Store=63c1038f18949b69fb052a65a9d44cbf/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:53:43,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:43,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:53:43,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:53:43,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 63c1038f18949b69fb052a65a9d44cbf; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72299099, jitterRate=0.07734052836894989}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:53:43,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:43,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., pid=11, masterSystemTime=1679622823109 2023-03-24 01:53:43,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:43,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:43,153 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=63c1038f18949b69fb052a65a9d44cbf, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:43,153 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622823152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622823152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622823152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622823152"}]},"ts":"1679622823152"} 2023-03-24 01:53:43,160 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-24 01:53:43,161 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 in 200 msec 2023-03-24 01:53:43,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-24 01:53:43,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, ASSIGN in 365 msec 2023-03-24 01:53:43,165 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:53:43,166 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622823166"}]},"ts":"1679622823166"} 2023-03-24 01:53:43,168 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:53:43,171 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:53:43,174 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 938 msec 2023-03-24 01:53:43,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 01:53:43,366 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-24 01:53:43,373 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x44ef43e3 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,378 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2678d495, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,381 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,385 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,388 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:53:43,394 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:53:43,402 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,407 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b886c8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,408 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,412 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,412 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,416 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,417 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,421 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,421 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,426 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,427 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,431 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,432 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,459 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,460 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,465 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,465 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:53:43,470 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:53:43,477 DEBUG [hconnection-0x38e5ca1f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,478 DEBUG [hconnection-0x186e1555-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,480 DEBUG [hconnection-0x7a95e8d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,481 DEBUG [hconnection-0x5ab397a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,482 DEBUG [hconnection-0xfa92e03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,483 DEBUG [hconnection-0x5eb08d54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,483 DEBUG [hconnection-0x37e8943a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,484 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,486 DEBUG [hconnection-0x27befc84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,486 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,489 DEBUG [hconnection-0x5229be5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:53:43,494 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48774, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,498 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48764, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,500 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48754, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,501 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48778, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,502 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48800, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,501 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:43,512 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48802, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:53:43,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:43,530 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:43,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:43,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:43,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:43,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:43,532 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:43,534 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:43,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:43,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:43,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:43,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:43,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:43,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:43,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:43,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:43,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:43,547 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:43,547 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:43,549 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:43,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:43,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:43,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:43,553 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:43,556 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 01:53:43,570 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:43,571 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:43,571 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:43,571 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:43,571 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:43,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:43,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:43,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622823654, max timestamp=1679622823746Num uniques 859; 2023-03-24 01:53:43,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:43,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622823654, max timestamp=1679622823746Num uniques 859; 2023-03-24 01:53:43,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622823654, max timestamp=1679622823750Num uniques 859; 2023-03-24 01:53:43,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:43,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622823789, max timestamp=1679622823812Num uniques 1018; 2023-03-24 01:53:43,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:53:43,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823789, max timestamp=1679622823812Num uniques 1018; 2023-03-24 01:53:43,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 01:53:43,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823789, max timestamp=1679622823812Num uniques 1018; 2023-03-24 01:53:43,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 01:53:43,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622823858, max timestamp=1679622823873Num uniques 1027; 2023-03-24 01:53:43,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 01:53:43,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622823858, max timestamp=1679622823874Num uniques 1009; 2023-03-24 01:53:43,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-24 01:53:43,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823858, max timestamp=1679622823873Num uniques 1009; 2023-03-24 01:53:43,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 01:53:43,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823915, max timestamp=1679622823925Num uniques 1009; 2023-03-24 01:53:43,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622823915, max timestamp=1679622823925Num uniques 1036; 2023-03-24 01:53:43,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 01:53:43,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 01:53:43,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823915, max timestamp=1679622823925Num uniques 1036; 2023-03-24 01:53:43,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-24 01:53:43,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:43,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622883977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:43,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7734, totalHeapSize=17816, min timestamp=1679622823953, max timestamp=1679622823982Num uniques 1009; 2023-03-24 01:53:43,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=7734, suffix heap size=132904,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=169 2023-03-24 01:53:43,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679622823953, max timestamp=1679622823981Num uniques 1009; 2023-03-24 01:53:43,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679622823953, max timestamp=1679622823982Num uniques 1045; 2023-03-24 01:53:43,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=8884, suffix heap size=132904,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=194 2023-03-24 01:53:43,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47855, new segment data size=7095, suffix heap size=136648,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=155 2023-03-24 01:53:44,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823993, max timestamp=1679622824019Num uniques 1014; 2023-03-24 01:53:44,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823993, max timestamp=1679622824019Num uniques 1028; 2023-03-24 01:53:44,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46440, new segment data size=6870, suffix heap size=133424,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=150 2023-03-24 01:53:44,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47084, new segment data size=6870, suffix heap size=134880,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=150 2023-03-24 01:53:44,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622823993, max timestamp=1679622824019Num uniques 1053; 2023-03-24 01:53:44,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=6870, suffix heap size=137480,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=150 2023-03-24 01:53:44,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622824047, max timestamp=1679622824053Num uniques 1009; 2023-03-24 01:53:44,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622824047, max timestamp=1679622824053Num uniques 1009; 2023-03-24 01:53:44,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-03-24 01:53:44,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-03-24 01:53:44,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622824047, max timestamp=1679622824053Num uniques 1009; 2023-03-24 01:53:44,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46208, new segment data size=9748, suffix heap size=132904,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=213 2023-03-24 01:53:44,130 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/805f4957f7484eb086aff87b598515d8 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679622824118, max timestamp=1679622824132Num uniques 1072; 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679622824118, max timestamp=1679622824132Num uniques 1072; 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622824118, max timestamp=1679622824132Num uniques 1072; 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=6870, suffix heap size=139456,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=150 2023-03-24 01:53:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 01:53:44,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679622824182, max timestamp=1679622824191Num uniques 1081; 2023-03-24 01:53:44,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-24 01:53:44,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622824182, max timestamp=1679622824191Num uniques 1009; 2023-03-24 01:53:44,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 01:53:44,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622824182, max timestamp=1679622824191Num uniques 1081; 2023-03-24 01:53:44,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=6870, suffix heap size=140392,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=150 2023-03-24 01:53:44,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679622824216, max timestamp=1679622824227Num uniques 1040; 2023-03-24 01:53:44,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=13280, suffix heap size=136128,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=290 2023-03-24 01:53:44,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679622824216, max timestamp=1679622824227Num uniques 1009; 2023-03-24 01:53:44,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13280, suffix heap size=132904,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=290 2023-03-24 01:53:44,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679622824216, max timestamp=1679622824227Num uniques 1009; 2023-03-24 01:53:44,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13280, suffix heap size=132904,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=290 2023-03-24 01:53:44,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679622824248, max timestamp=1679622824263Num uniques 1149; 2023-03-24 01:53:44,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-24 01:53:44,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679622824248, max timestamp=1679622824263Num uniques 1149; 2023-03-24 01:53:44,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679622824248, max timestamp=1679622824263Num uniques 1149; 2023-03-24 01:53:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-24 01:53:44,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-03-24 01:53:44,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679622824282, max timestamp=1679622824290Num uniques 1108; 2023-03-24 01:53:44,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 01:53:44,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679622824282, max timestamp=1679622824291Num uniques 1108; 2023-03-24 01:53:44,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 01:53:44,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679622824282, max timestamp=1679622824291Num uniques 1108; 2023-03-24 01:53:44,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 01:53:44,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679622824310, max timestamp=1679622824317Num uniques 1017; 2023-03-24 01:53:44,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679622824310, max timestamp=1679622824317Num uniques 1017; 2023-03-24 01:53:44,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679622824310, max timestamp=1679622824317Num uniques 1017; 2023-03-24 01:53:44,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-24 01:53:44,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-24 01:53:44,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-24 01:53:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679622824330, max timestamp=1679622824345Num uniques 1026; 2023-03-24 01:53:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-24 01:53:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679622824330, max timestamp=1679622824343Num uniques 1026; 2023-03-24 01:53:44,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-24 01:53:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679622824330, max timestamp=1679622824345Num uniques 1026; 2023-03-24 01:53:44,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-03-24 01:53:44,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10750, totalHeapSize=24680, min timestamp=1679622824362, max timestamp=1679622824368Num uniques 1135; 2023-03-24 01:53:44,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51970, new segment data size=10750, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-24 01:53:44,371 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10759, totalHeapSize=24680, min timestamp=1679622824362, max timestamp=1679622824368Num uniques 1135; 2023-03-24 01:53:44,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51979, new segment data size=10759, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-24 01:53:44,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1679622824362, max timestamp=1679622824368Num uniques 1135; 2023-03-24 01:53:44,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51970, new segment data size=9518, suffix heap size=146008,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=208 2023-03-24 01:53:44,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622884419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1679622824417, max timestamp=1679622824421Num uniques 1094; 2023-03-24 01:53:44,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50094, new segment data size=8874, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 01:53:44,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=182, cellSize=8323, totalHeapSize=19168, min timestamp=1679622824417, max timestamp=1679622824421Num uniques 1067; 2023-03-24 01:53:44,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48853, new segment data size=8323, suffix heap size=138936,new segment heap size=19168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=182 2023-03-24 01:53:44,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679622824417, max timestamp=1679622824421Num uniques 1094; 2023-03-24 01:53:44,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 01:53:44,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679622824439, max timestamp=1679622824448Num uniques 1053; 2023-03-24 01:53:44,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679622824439, max timestamp=1679622824448Num uniques 1053; 2023-03-24 01:53:44,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 01:53:44,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48219, new segment data size=8884, suffix heap size=137480,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=194 2023-03-24 01:53:44,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679622824439, max timestamp=1679622824448Num uniques 1041; 2023-03-24 01:53:44,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47683, new segment data size=7005, suffix heap size=136232,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1041, new segment cells count=153 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679622824463, max timestamp=1679622824473Num uniques 1053; 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679622824463, max timestamp=1679622824473Num uniques 1012; 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1679622824463, max timestamp=1679622824473Num uniques 1012; 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-03-24 01:53:44,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48227, new segment data size=11992, suffix heap size=137480,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=262 2023-03-24 01:53:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679622824496, max timestamp=1679622824503Num uniques 1121; 2023-03-24 01:53:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-24 01:53:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679622824496, max timestamp=1679622824503Num uniques 1121; 2023-03-24 01:53:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-24 01:53:44,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679622824496, max timestamp=1679622824503Num uniques 1121; 2023-03-24 01:53:44,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=380, cellSize=17400, totalHeapSize=39760, min timestamp=1679622824522, max timestamp=1679622824534Num uniques 1030; 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=380, cellSize=17392, totalHeapSize=39760, min timestamp=1679622824522, max timestamp=1679622824534Num uniques 1030; 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=380, cellSize=17392, totalHeapSize=39760, min timestamp=1679622824522, max timestamp=1679622824534Num uniques 1030; 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=17400, suffix heap size=135088,new segment heap size=39760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=380 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47162, new segment data size=17392, suffix heap size=135088,new segment heap size=39760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=380 2023-03-24 01:53:44,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47162, new segment data size=17392, suffix heap size=135088,new segment heap size=39760 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=380 2023-03-24 01:53:44,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622884579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622884580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622884581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622884582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622884583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,680 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/8c27f6bb491a42fcbe49377ce6e11a5c 2023-03-24 01:53:44,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622884687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622884687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622884688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622884690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622884690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,776 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/e445c2bba29d4c91bf3e8752eab93430 2023-03-24 01:53:44,788 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/805f4957f7484eb086aff87b598515d8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8 2023-03-24 01:53:44,799 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8, entries=150, sequenceid=8, filesize=11.6 K 2023-03-24 01:53:44,802 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/8c27f6bb491a42fcbe49377ce6e11a5c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c 2023-03-24 01:53:44,817 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c, entries=150, sequenceid=8, filesize=11.6 K 2023-03-24 01:53:44,823 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/e445c2bba29d4c91bf3e8752eab93430 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430 2023-03-24 01:53:44,836 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430, entries=150, sequenceid=8, filesize=11.6 K 2023-03-24 01:53:44,839 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 63c1038f18949b69fb052a65a9d44cbf in 1283ms, sequenceid=8, compaction requested=false 2023-03-24 01:53:44,841 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 01:53:44,843 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:44,843 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:44,844 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:44,844 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:44,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:44,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:44,856 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8fd64a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:44,856 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,858 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:44,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:44,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:44,881 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:44,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:44,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:44,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:44,881 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:44,889 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:44,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:44,892 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,899 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:44,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:44,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:44,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:44,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:44,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:44,920 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,920 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:44,920 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:44,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:44,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:44,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:44,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679622824578, max timestamp=1679622824929Num uniques 1239; 2023-03-24 01:53:44,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56744, new segment data size=8654, suffix heap size=156824,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1239, new segment cells count=189 2023-03-24 01:53:44,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:44,946 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=120.76 KB heapSize=330.86 KB 2023-03-24 01:53:44,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679622824578, max timestamp=1679622824929Num uniques 1239; 2023-03-24 01:53:44,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56744, new segment data size=8654, suffix heap size=156824,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1239, new segment cells count=189 2023-03-24 01:53:44,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679622824578, max timestamp=1679622824933Num uniques 1239; 2023-03-24 01:53:44,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=56744, new segment data size=8654, suffix heap size=156824,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1239, new segment cells count=189 2023-03-24 01:53:44,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:44,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:44,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:44,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:44,954 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:44,954 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:45,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:45,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:45,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:45,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:45,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:45,048 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:45,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:45,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:45,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,053 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:45,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:45,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:45,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:45,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:45,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:45,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:45,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,090 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:45,090 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:45,090 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:45,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679622825047, max timestamp=1679622825087Num uniques 859; 2023-03-24 01:53:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 01:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679622825047, max timestamp=1679622825087Num uniques 859; 2023-03-24 01:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 01:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679622825047, max timestamp=1679622825087Num uniques 859; 2023-03-24 01:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 01:53:45,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,100 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,100 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a601af1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:45,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:45,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:45,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:45,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:45,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:45,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:45,117 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:45,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:45,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:45,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:45,117 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:45,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,123 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,125 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,127 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:45,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:45,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:45,139 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:45,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:45,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622885146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622885147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622885147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622885149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622885152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:45,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:45,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:45,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622885254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622885255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622885256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622885257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622885258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:45,285 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:45,285 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:45,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:45,295 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,295 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,297 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:45,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:45,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:45,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:45,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:45,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:45,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:45,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:45,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:45,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:45,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:45,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:45,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:45,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c271bea[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:45,356 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:45,356 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,358 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:45,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:45,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:45,409 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:45,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:45,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:45,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:45,409 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:45,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,418 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/32ccd25caf1040c28cc8e30fd6441c86 2023-03-24 01:53:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:45,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:45,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:45,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:45,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:45,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622885475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622885475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622885493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622885494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622885494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,593 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:45,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:45,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:45,608 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:45,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:45,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:45,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:45,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:45,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:45,618 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:45,618 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:45,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:45,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:45,624 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,624 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:45,624 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:45,624 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:45,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,629 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:45,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b9612[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:45,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:45,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:45,638 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,638 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:45,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:45,663 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:45,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:45,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:45,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:45,663 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:45,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,671 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:45,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:45,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:45,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:45,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,791 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:45,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:45,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622885797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:45,811 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:45,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:45,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:45,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,815 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622885813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622885813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622885813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:45,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622885813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:45,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:45,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:45,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:45,823 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:45,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:45,826 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,826 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63522d2f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:45,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:45,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:45,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:45,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:45,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:45,851 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:45,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:45,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:45,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:45,852 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:45,854 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,854 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,854 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,856 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:45,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:45,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:45,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:45,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:45,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:45,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:45,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:45,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:45,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:45,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:45,870 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:45,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:45,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:45,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1ab71d1290fe427abb03026052a2559d 2023-03-24 01:53:45,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:45,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:45,975 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:53:45,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:46,004 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:46,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:46,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:46,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,012 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,012 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:46,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:46,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:46,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:46,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:46,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:46,032 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,032 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:46,032 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:46,033 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:46,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:46,045 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,048 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ff1426[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:46,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:46,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:46,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:46,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:46,081 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:46,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:46,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:46,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:46,081 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:46,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,086 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:46,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:46,094 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:46,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:46,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,097 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:46,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,121 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-24 01:53:46,124 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-24 01:53:46,197 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:46,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:46,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:46,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:46,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:46,208 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:46,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:46,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:46,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,217 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:46,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:46,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:46,229 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,229 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:46,229 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:46,229 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:46,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d53b8b8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:46,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:46,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:46,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:46,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:46,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:46,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:46,257 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:46,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:46,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:46,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:46,257 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:46,262 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,262 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,265 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:46,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,273 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:46,273 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:46,273 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,273 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,273 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:46,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:46,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622886304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622886319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:46,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622886319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622886319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:46,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622886320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,373 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:46,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:46,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:46,380 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:46,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:46,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:46,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,384 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,385 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,385 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:46,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:46,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:46,392 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,392 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:46,392 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:46,392 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:46,395 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76339006[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:46,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:46,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,400 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,400 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:46,400 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:46,400 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:46,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:46,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:46,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,404 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:46,412 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:46,412 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:46,412 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:46,412 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:46,412 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:46,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,418 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:46,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,425 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,426 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:46,426 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:46,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/dd817216eb45424b81a8b4d3b061c269 2023-03-24 01:53:46,461 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 01:53:46,462 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-24 01:53:46,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/32ccd25caf1040c28cc8e30fd6441c86 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86 2023-03-24 01:53:46,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86, entries=150, sequenceid=374, filesize=11.9 K 2023-03-24 01:53:46,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1ab71d1290fe427abb03026052a2559d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d 2023-03-24 01:53:46,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d, entries=150, sequenceid=374, filesize=11.9 K 2023-03-24 01:53:46,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/dd817216eb45424b81a8b4d3b061c269 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269 2023-03-24 01:53:46,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269, entries=150, sequenceid=374, filesize=11.9 K 2023-03-24 01:53:46,525 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~103.22 KB/105696, currentSize=154.31 KB/158010 for 63c1038f18949b69fb052a65a9d44cbf in 1578ms, sequenceid=374, compaction requested=false 2023-03-24 01:53:46,525 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:46,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:46,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:46,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:46,533 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:46,533 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:46,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:46,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:46,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:46,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:46,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:46,552 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=154.31 KB heapSize=416.63 KB 2023-03-24 01:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:46,553 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:46,553 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:46,553 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:46,553 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:46,616 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/89115f5c98e247e8b9861f8ba5f98fa3 2023-03-24 01:53:46,664 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/438cd393faf64ca7bc4a26f5cb09fd96 2023-03-24 01:53:46,702 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/6a4d5160f7fa428382e9dfc3851887bb 2023-03-24 01:53:46,717 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/89115f5c98e247e8b9861f8ba5f98fa3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3 2023-03-24 01:53:46,733 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3, entries=150, sequenceid=408, filesize=11.9 K 2023-03-24 01:53:46,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/438cd393faf64ca7bc4a26f5cb09fd96 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96 2023-03-24 01:53:46,751 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96, entries=150, sequenceid=408, filesize=11.9 K 2023-03-24 01:53:46,755 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/6a4d5160f7fa428382e9dfc3851887bb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb 2023-03-24 01:53:46,778 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb, entries=150, sequenceid=408, filesize=11.9 K 2023-03-24 01:53:46,781 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~416.58 KB/426576, currentSize=0 B/0 for 63c1038f18949b69fb052a65a9d44cbf in 229ms, sequenceid=408, compaction requested=true 2023-03-24 01:53:46,781 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:46,782 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:46,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:46,787 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ae0ba[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:46,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:46,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,792 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:46,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:46,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:46,795 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,796 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:46,795 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:46,805 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:46,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:46,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:46,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:46,805 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:46,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:46,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:46,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:46,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:46,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:46,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:46,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:46,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:46,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:46,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:46,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:46,965 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:46,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,967 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:46,967 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:46,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:46,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:46,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:46,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:46,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:46,988 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,988 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:46,988 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:46,988 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:46,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:46,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:46,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5258a37[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:46,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:46,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:46,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:46,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:46,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:46,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:46,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:47,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:47,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:47,020 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:47,020 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:47,020 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:47,020 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:47,020 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:47,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,026 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,033 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:47,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:47,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:47,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:47,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,034 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:47,034 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:47,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:47,134 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:47,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:47,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:47,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:47,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:47,142 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:47,142 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:47,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:47,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:47,146 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,146 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:47,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:47,152 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,153 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,153 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:47,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:47,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:47,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:47,160 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:47,164 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:47,165 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:47,165 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:47,165 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:47,165 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,165 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:47,166 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:47,169 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:47,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35837453[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:47,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,175 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:47,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:47,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:47,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:47,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:47,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:47,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:47,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:47,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:47,181 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,184 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:47,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:47,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,193 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,193 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:47,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:47,193 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:47,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,194 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:47,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:47,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,194 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:47,295 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:47,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:47,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:47,308 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:47,308 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:47,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:47,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:47,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,312 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:47,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:47,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:47,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:47,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:47,325 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:47,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:47,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:47,326 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:47,326 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:47,326 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 01:53:47,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:47,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:47,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:47,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:47,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:47,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:47,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:47,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:47,381 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/9a8303f09e5e4a418f8995d48bcffb2a 2023-03-24 01:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622827386, max timestamp=1679622827393Num uniques 859; 2023-03-24 01:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622827386, max timestamp=1679622827393Num uniques 859; 2023-03-24 01:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622827386, max timestamp=1679622827393Num uniques 859; 2023-03-24 01:53:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 01:53:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:47,426 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/5e4d38a627ce43d1a13aed5bad215808 2023-03-24 01:53:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679622827412, max timestamp=1679622827433Num uniques 1018; 2023-03-24 01:53:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679622827412, max timestamp=1679622827433Num uniques 1018; 2023-03-24 01:53:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-24 01:53:47,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-24 01:53:47,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622827426, max timestamp=1679622827433Num uniques 1009; 2023-03-24 01:53:47,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-03-24 01:53:47,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1679622827453, max timestamp=1679622827470Num uniques 1177; 2023-03-24 01:53:47,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53893, new segment data size=8093, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 01:53:47,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622827453, max timestamp=1679622827470Num uniques 1177; 2023-03-24 01:53:47,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 01:53:47,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1679622827453, max timestamp=1679622827470Num uniques 1077; 2023-03-24 01:53:47,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-24 01:53:47,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622827498, max timestamp=1679622827508Num uniques 1036; 2023-03-24 01:53:47,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 01:53:47,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622827498, max timestamp=1679622827508Num uniques 1036; 2023-03-24 01:53:47,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 01:53:47,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622827498, max timestamp=1679622827508Num uniques 1036; 2023-03-24 01:53:47,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 01:53:47,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18081, totalHeapSize=41320, min timestamp=1679622827521, max timestamp=1679622827534Num uniques 1045; 2023-03-24 01:53:47,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47851, new segment data size=18081, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-24 01:53:47,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18080, totalHeapSize=41320, min timestamp=1679622827521, max timestamp=1679622827534Num uniques 1045; 2023-03-24 01:53:47,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=18080, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-24 01:53:47,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18080, totalHeapSize=41320, min timestamp=1679622827521, max timestamp=1679622827534Num uniques 1045; 2023-03-24 01:53:47,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=18080, suffix heap size=136648,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=395 2023-03-24 01:53:47,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23074, totalHeapSize=52656, min timestamp=1679622827534, max timestamp=1679622827557Num uniques 1254; 2023-03-24 01:53:47,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23074, totalHeapSize=52656, min timestamp=1679622827534, max timestamp=1679622827557Num uniques 1254; 2023-03-24 01:53:47,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57424, new segment data size=23074, suffix heap size=158384,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=504 2023-03-24 01:53:47,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=57424, new segment data size=23074, suffix heap size=158384,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=504 2023-03-24 01:53:47,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23079, totalHeapSize=52656, min timestamp=1679622827534, max timestamp=1679622827557Num uniques 1254; 2023-03-24 01:53:47,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=57429, new segment data size=23079, suffix heap size=158384,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=504 2023-03-24 01:53:47,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622887580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622887581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622887582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622887584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48746 deadline: 1679622887585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48794 deadline: 1679622887687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622887688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622887690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622887692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622887694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622887891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,894 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/5dc5cf4c55b54bd98b788de678bb1662 2023-03-24 01:53:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622887894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48774 deadline: 1679622887895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622887901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622887895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,917 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/9a8303f09e5e4a418f8995d48bcffb2a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a 2023-03-24 01:53:47,932 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a, entries=100, sequenceid=415, filesize=9.5 K 2023-03-24 01:53:47,935 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/5e4d38a627ce43d1a13aed5bad215808 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808 2023-03-24 01:53:47,951 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808, entries=100, sequenceid=415, filesize=9.5 K 2023-03-24 01:53:47,955 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/5dc5cf4c55b54bd98b788de678bb1662 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662 2023-03-24 01:53:47,964 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662, entries=100, sequenceid=415, filesize=9.5 K 2023-03-24 01:53:47,966 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=181.14 KB/185490 for 63c1038f18949b69fb052a65a9d44cbf in 640ms, sequenceid=415, compaction requested=true 2023-03-24 01:53:47,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:47,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:47,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:47,967 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:47,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5a7145[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:47,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:47,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,978 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:47,979 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:47,979 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,979 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:47,980 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:47,984 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:47,984 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:47,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:47,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:47,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:47,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:47,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:47,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:47,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:47,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:47,992 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:47,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:47,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:47,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:47,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:47,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:47,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:47,994 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:47,994 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:47,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:47,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:48,094 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:48,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:48,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:48,100 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:48,100 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:48,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:48,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:48,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,104 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:48,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:48,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:48,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:48,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:48,109 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:48,109 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:48,109 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:48,110 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=181.14 KB heapSize=492.09 KB 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:48,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:48,129 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/2ff03e2ca0304c71933b4f6341423f00 2023-03-24 01:53:48,158 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/ac4e787dfd324e8093f4595cf3006bd3 2023-03-24 01:53:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:48,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:48,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622888199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48754 deadline: 1679622888204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622888205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622888206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48802 deadline: 1679622888208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622888304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622888516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,590 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/9e96bd741b3d46b4b25c62a86744d840 2023-03-24 01:53:48,602 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/2ff03e2ca0304c71933b4f6341423f00 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00 2023-03-24 01:53:48,612 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00, entries=150, sequenceid=538, filesize=11.9 K 2023-03-24 01:53:48,614 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/ac4e787dfd324e8093f4595cf3006bd3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3 2023-03-24 01:53:48,642 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3, entries=150, sequenceid=538, filesize=11.9 K 2023-03-24 01:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/9e96bd741b3d46b4b25c62a86744d840 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840 2023-03-24 01:53:48,673 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840, entries=150, sequenceid=538, filesize=11.9 K 2023-03-24 01:53:48,675 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~181.14 KB/185490, heapSize ~492.05 KB/503856, currentSize=6.71 KB/6870 for 63c1038f18949b69fb052a65a9d44cbf in 565ms, sequenceid=538, compaction requested=true 2023-03-24 01:53:48,679 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:48,679 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:48,680 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,682 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c762524[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:48,686 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:48,686 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:48,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:48,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:48,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:48,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:48,694 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:48,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:48,694 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:48,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:48,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:48,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:48,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:48,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:48,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:48,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:48,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:48,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:48,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:48,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,707 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,707 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,708 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:48,708 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:48,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:48,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:48,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:48,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/639603e6f00e42159818e91328c07be0 2023-03-24 01:53:48,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622828750, max timestamp=1679622828779Num uniques 859; 2023-03-24 01:53:48,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:53:48,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622828750, max timestamp=1679622828779Num uniques 859; 2023-03-24 01:53:48,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:53:48,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622828750, max timestamp=1679622828779Num uniques 859; 2023-03-24 01:53:48,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:53:48,808 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:48,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:48,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:48,831 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:48,831 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:48,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:48,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:48,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,841 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622888815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,841 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:48,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622888817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/c1e418f4582046c99da248cb0f25508e 2023-03-24 01:53:48,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:48,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:48,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:48,846 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:48,846 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:48,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:48,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622888820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622888837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622888840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,862 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:48,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:48,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:48,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:48,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:48,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ab72af[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:48,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:48,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,885 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:48,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:48,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:48,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:48,887 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:48,894 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:48,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:48,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:48,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:48,894 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:48,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:48,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,913 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:48,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/c6bb879aab6044d985f3f443f08a5d20 2023-03-24 01:53:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:48,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:48,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/639603e6f00e42159818e91328c07be0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0 2023-03-24 01:53:48,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:48,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:48,929 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:48,929 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:48,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:48,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:48,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:48,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:48,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:48,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:48,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:48,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:48,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0, entries=250, sequenceid=548, filesize=16.7 K 2023-03-24 01:53:48,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/c1e418f4582046c99da248cb0f25508e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e 2023-03-24 01:53:48,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622888944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622888951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e, entries=150, sequenceid=548, filesize=11.9 K 2023-03-24 01:53:48,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/c6bb879aab6044d985f3f443f08a5d20 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20 2023-03-24 01:53:48,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622888962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622888966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:48,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20, entries=150, sequenceid=548, filesize=11.9 K 2023-03-24 01:53:49,000 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 63c1038f18949b69fb052a65a9d44cbf in 277ms, sequenceid=548, compaction requested=true 2023-03-24 01:53:49,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:49,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:49,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:49,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:49,003 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:49,003 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:49,017 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70372 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:49,017 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75252 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:49,019 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:53:49,019 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:49,020 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=68.7 K 2023-03-24 01:53:49,019 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:53:49,020 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:49,020 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=73.5 K 2023-03-24 01:53:49,022 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 805f4957f7484eb086aff87b598515d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679622823517 2023-03-24 01:53:49,022 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8c27f6bb491a42fcbe49377ce6e11a5c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679622823517 2023-03-24 01:53:49,026 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 32ccd25caf1040c28cc8e30fd6441c86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679622824580 2023-03-24 01:53:49,027 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 89115f5c98e247e8b9861f8ba5f98fa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622825113 2023-03-24 01:53:49,028 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1ab71d1290fe427abb03026052a2559d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679622824580 2023-03-24 01:53:49,029 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9a8303f09e5e4a418f8995d48bcffb2a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679622827313 2023-03-24 01:53:49,029 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 438cd393faf64ca7bc4a26f5cb09fd96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622825113 2023-03-24 01:53:49,029 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2ff03e2ca0304c71933b4f6341423f00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1679622827579 2023-03-24 01:53:49,029 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:49,030 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5e4d38a627ce43d1a13aed5bad215808, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679622827313 2023-03-24 01:53:49,030 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 639603e6f00e42159818e91328c07be0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828711 2023-03-24 01:53:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:49,031 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ac4e787dfd324e8093f4595cf3006bd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1679622827579 2023-03-24 01:53:49,033 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c1e418f4582046c99da248cb0f25508e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828714 2023-03-24 01:53:49,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:49,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:49,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:49,049 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:49,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:49,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:49,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:49,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:49,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:49,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:49,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:49,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:49,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:49,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:49,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:49,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:49,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:49,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:49,071 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:49,071 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:49,142 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#19 average throughput is 0.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:49,162 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#20 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:49,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:49,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:49,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622889197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622889201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622889202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,227 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/51a0c624d6f24445b8bccd91b523b50a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/51a0c624d6f24445b8bccd91b523b50a 2023-03-24 01:53:49,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622889203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,248 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/6059547a18014e24aca78e317c96c8a2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6059547a18014e24aca78e317c96c8a2 2023-03-24 01:53:49,259 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into 51a0c624d6f24445b8bccd91b523b50a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:49,259 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into 6059547a18014e24aca78e317c96c8a2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:49,259 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:49,260 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:49,260 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=10, startTime=1679622829002; duration=0sec 2023-03-24 01:53:49,260 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=10, startTime=1679622829002; duration=0sec 2023-03-24 01:53:49,260 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:49,260 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:49,261 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:49,265 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70372 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:49,266 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:53:49,266 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:49,266 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=68.7 K 2023-03-24 01:53:49,269 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e445c2bba29d4c91bf3e8752eab93430, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679622823517 2023-03-24 01:53:49,272 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting dd817216eb45424b81a8b4d3b061c269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679622824580 2023-03-24 01:53:49,273 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6a4d5160f7fa428382e9dfc3851887bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622825113 2023-03-24 01:53:49,279 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5dc5cf4c55b54bd98b788de678bb1662, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679622827313 2023-03-24 01:53:49,282 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9e96bd741b3d46b4b25c62a86744d840, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1679622827579 2023-03-24 01:53:49,283 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c6bb879aab6044d985f3f443f08a5d20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828714 2023-03-24 01:53:49,316 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:49,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622889315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622889315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622889315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622889338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622889368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622889527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622889530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622889531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622889544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,587 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/7527c006a2884a469f6dfd287a4c837c 2023-03-24 01:53:49,617 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/2c45573dd9724b12baf29630cb6d4455 2023-03-24 01:53:49,767 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/492ae215a4cc487f992658d16824ee24 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/492ae215a4cc487f992658d16824ee24 2023-03-24 01:53:49,779 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 492ae215a4cc487f992658d16824ee24(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:49,779 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:49,779 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=10, startTime=1679622829002; duration=0sec 2023-03-24 01:53:49,781 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622889832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622889832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622889838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622889848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,071 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/bb3d25f5275548738940515ab2504909 2023-03-24 01:53:50,092 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/7527c006a2884a469f6dfd287a4c837c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c 2023-03-24 01:53:50,106 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 01:53:50,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/2c45573dd9724b12baf29630cb6d4455 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455 2023-03-24 01:53:50,134 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 01:53:50,137 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/bb3d25f5275548738940515ab2504909 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909 2023-03-24 01:53:50,159 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 01:53:50,164 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 63c1038f18949b69fb052a65a9d44cbf in 1094ms, sequenceid=583, compaction requested=false 2023-03-24 01:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:50,170 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b24bba5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:50,176 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:50,176 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:50,181 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:50,183 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:50,183 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:50,190 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:50,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:50,190 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:50,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:50,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:50,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:50,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:50,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:50,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:50,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:50,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:50,215 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:50,215 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:50,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:50,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:50,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:50,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:50,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:50,321 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:50,321 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:50,322 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:50,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:50,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:50,326 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:50,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:50,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:50,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:50,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:50,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:50,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:50,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:50,332 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:50,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:50,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:50,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:50,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:50,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:50,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:50,336 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:50,359 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/f13793f53e4f4923bbdd4adb7c260f3d 2023-03-24 01:53:50,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622830371, max timestamp=1679622830375Num uniques 859; 2023-03-24 01:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622830371, max timestamp=1679622830375Num uniques 859; 2023-03-24 01:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622830371, max timestamp=1679622830375Num uniques 859; 2023-03-24 01:53:50,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:53:50,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622890396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622890403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622890404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622890409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622890409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622890510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622890510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622890514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622890514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622890514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622890717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622890717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622890718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622890718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622890719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:50,802 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/54e8dfde0b0e4c23ae272cf2fe6156b9 2023-03-24 01:53:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622891022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622891023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622891023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622891024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622891025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,223 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=596 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/cb45e44cc0f6493eae54b34055cedcfc 2023-03-24 01:53:51,240 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/f13793f53e4f4923bbdd4adb7c260f3d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d 2023-03-24 01:53:51,249 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d, entries=150, sequenceid=596, filesize=11.9 K 2023-03-24 01:53:51,252 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/54e8dfde0b0e4c23ae272cf2fe6156b9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9 2023-03-24 01:53:51,264 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9, entries=150, sequenceid=596, filesize=11.9 K 2023-03-24 01:53:51,265 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/cb45e44cc0f6493eae54b34055cedcfc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc 2023-03-24 01:53:51,282 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc, entries=150, sequenceid=596, filesize=11.9 K 2023-03-24 01:53:51,284 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 63c1038f18949b69fb052a65a9d44cbf in 952ms, sequenceid=596, compaction requested=true 2023-03-24 01:53:51,284 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:51,284 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:51,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f823dd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:51,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:51,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:51,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:51,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:51,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:51,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:51,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:51,311 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:51,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:51,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:51,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:51,311 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:51,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:51,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:51,315 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,317 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:51,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:51,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:51,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:51,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:51,331 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:51,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,333 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:51,333 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:51,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:51,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:51,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:51,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:51,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:51,441 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:51,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:51,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:51,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:51,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:51,447 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:51,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:51,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:51,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:51,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:51,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:51,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:51,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:51,457 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:51,457 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:51,457 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:51,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:51,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:51,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:51,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622891537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622891540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622891541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622891541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622891546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622891643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622891648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622891648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622891648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622891648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622891855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622891855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622891856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622891857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:51,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622891866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:51,877 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/d008c2825d9f4428ab806ac6be0e6679 2023-03-24 01:53:52,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622892158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622892160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622892160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622892162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622892169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,306 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/3957caf72d4b4ea494993bc852a69cf7 2023-03-24 01:53:52,361 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/4da25771a15e47d380363fd998446286 2023-03-24 01:53:52,369 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/d008c2825d9f4428ab806ac6be0e6679 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679 2023-03-24 01:53:52,377 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679, entries=150, sequenceid=633, filesize=11.9 K 2023-03-24 01:53:52,388 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/3957caf72d4b4ea494993bc852a69cf7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7 2023-03-24 01:53:52,398 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7, entries=150, sequenceid=633, filesize=11.9 K 2023-03-24 01:53:52,402 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/4da25771a15e47d380363fd998446286 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286 2023-03-24 01:53:52,411 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286, entries=150, sequenceid=633, filesize=11.9 K 2023-03-24 01:53:52,413 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 63c1038f18949b69fb052a65a9d44cbf in 956ms, sequenceid=633, compaction requested=true 2023-03-24 01:53:52,414 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:52,414 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:52,414 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:52,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:52,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:52,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:52,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:52,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a9a6903[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:52,424 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:52,424 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:52,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:52,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:52,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:52,440 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:52,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:52,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:52,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:52,440 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:52,442 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,442 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:52,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,442 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:52,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:52,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:52,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:52,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:52,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:52,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:52,454 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:52,454 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:52,554 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:52,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:52,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:52,559 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:52,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:52,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:52,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:52,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:52,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:52,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:52,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:52,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:52,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:52,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:52,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:52,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:52,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:52,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:52,582 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:52,582 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:52,582 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:52,583 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:52,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:52,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:52,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622832686, max timestamp=1679622832696Num uniques 859; 2023-03-24 01:53:52,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:52,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622832686, max timestamp=1679622832696Num uniques 859; 2023-03-24 01:53:52,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:52,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622832686, max timestamp=1679622832696Num uniques 859; 2023-03-24 01:53:52,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:52,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622832725, max timestamp=1679622832734Num uniques 1018; 2023-03-24 01:53:52,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622832725, max timestamp=1679622832734Num uniques 1018; 2023-03-24 01:53:52,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622832725, max timestamp=1679622832734Num uniques 1018; 2023-03-24 01:53:52,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:53:52,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:53:52,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 01:53:52,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622892773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679622832770, max timestamp=1679622832775Num uniques 1027; 2023-03-24 01:53:52,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-03-24 01:53:52,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622892773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622832771, max timestamp=1679622832775Num uniques 1009; 2023-03-24 01:53:52,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-24 01:53:52,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622832771, max timestamp=1679622832775Num uniques 1027; 2023-03-24 01:53:52,780 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 01:53:52,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622892800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622892801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622892805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622892881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622892881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622892910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622892910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:52,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:52,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622892915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,019 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/b8e845084eaf4d8bb52b4ec93570640e 2023-03-24 01:53:53,045 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1b75a7d9605d45ddaf400402be1ec46e 2023-03-24 01:53:53,063 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=642 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/25974c6b232d4e51bddf4093b644234e 2023-03-24 01:53:53,085 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/b8e845084eaf4d8bb52b4ec93570640e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e 2023-03-24 01:53:53,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622893089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622893090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,094 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e, entries=150, sequenceid=642, filesize=11.9 K 2023-03-24 01:53:53,100 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1b75a7d9605d45ddaf400402be1ec46e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e 2023-03-24 01:53:53,107 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e, entries=150, sequenceid=642, filesize=11.9 K 2023-03-24 01:53:53,110 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/25974c6b232d4e51bddf4093b644234e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e 2023-03-24 01:53:53,117 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e, entries=150, sequenceid=642, filesize=11.9 K 2023-03-24 01:53:53,121 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 63c1038f18949b69fb052a65a9d44cbf in 538ms, sequenceid=642, compaction requested=true 2023-03-24 01:53:53,121 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:53,121 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:53,125 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=147.60 KB heapSize=419.16 KB 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:53,126 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 01:53:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:53,127 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da4af5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:53,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:53,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,136 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:53,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:53,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:53,148 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:53,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:53,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:53,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:53,148 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:53,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/b527642ecbdb4ddebe68008c767c54f4 2023-03-24 01:53:53,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,160 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,162 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622893170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622893172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622893179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:53,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:53,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:53,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:53,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:53,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:53,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:53,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:53,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:53,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:53,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622893281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622893282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622893284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:53,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:53,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:53,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:53,318 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:53,318 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:53,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:53,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:53,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,371 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:53,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:53,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:53,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:53,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:53,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:53,380 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,380 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:53,381 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:53,381 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:53,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:53,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622893391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622893393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,424 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,425 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d3858[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:53,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:53,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:53,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:53,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622893485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622893486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622893490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,509 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,509 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,509 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:53,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:53,513 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:53,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:53,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:53,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:53,514 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:53,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:53,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,558 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/db7b54ddd1ab440690c23c9188d0de32 2023-03-24 01:53:53,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/f00bee47d43f4328b5ab9b5f1cabcca0 2023-03-24 01:53:53,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/b527642ecbdb4ddebe68008c767c54f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4 2023-03-24 01:53:53,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4, entries=200, sequenceid=714, filesize=14.3 K 2023-03-24 01:53:53,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,602 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:53,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:53,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:53,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:53,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:53,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:53,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/db7b54ddd1ab440690c23c9188d0de32 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32 2023-03-24 01:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:53,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:53,607 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:53,607 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:53,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32, entries=150, sequenceid=714, filesize=11.9 K 2023-03-24 01:53:53,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/f00bee47d43f4328b5ab9b5f1cabcca0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0 2023-03-24 01:53:53,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0, entries=150, sequenceid=714, filesize=11.9 K 2023-03-24 01:53:53,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.11 KB/429168, currentSize=33.54 KB/34350 for 63c1038f18949b69fb052a65a9d44cbf in 499ms, sequenceid=714, compaction requested=true 2023-03-24 01:53:53,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:53,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:53,625 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:53,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:53,625 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:53:53,625 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:53,629 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75762 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:53,629 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:53:53,629 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,629 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/51a0c624d6f24445b8bccd91b523b50a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=74.0 K 2023-03-24 01:53:53,629 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73322 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:53,629 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:53:53,629 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,629 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6059547a18014e24aca78e317c96c8a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=71.6 K 2023-03-24 01:53:53,630 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 51a0c624d6f24445b8bccd91b523b50a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828714 2023-03-24 01:53:53,631 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6059547a18014e24aca78e317c96c8a2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828714 2023-03-24 01:53:53,632 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7527c006a2884a469f6dfd287a4c837c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679622828802 2023-03-24 01:53:53,633 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2c45573dd9724b12baf29630cb6d4455, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679622828802 2023-03-24 01:53:53,634 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f13793f53e4f4923bbdd4adb7c260f3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1679622829194 2023-03-24 01:53:53,634 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d008c2825d9f4428ab806ac6be0e6679, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679622830398 2023-03-24 01:53:53,634 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 54e8dfde0b0e4c23ae272cf2fe6156b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1679622829194 2023-03-24 01:53:53,635 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b8e845084eaf4d8bb52b4ec93570640e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1679622831530 2023-03-24 01:53:53,635 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3957caf72d4b4ea494993bc852a69cf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679622830398 2023-03-24 01:53:53,635 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b527642ecbdb4ddebe68008c767c54f4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:53,636 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1b75a7d9605d45ddaf400402be1ec46e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1679622831530 2023-03-24 01:53:53,637 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting db7b54ddd1ab440690c23c9188d0de32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:53,654 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:53,658 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:53,677 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/90b20b0f11ba40269ee885747487c413 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/90b20b0f11ba40269ee885747487c413 2023-03-24 01:53:53,678 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/d7e72572f54649c39f98d78ec377d43c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d7e72572f54649c39f98d78ec377d43c 2023-03-24 01:53:53,685 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into 90b20b0f11ba40269ee885747487c413(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:53,686 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:53,686 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=10, startTime=1679622833625; duration=0sec 2023-03-24 01:53:53,686 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:53,686 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into d7e72572f54649c39f98d78ec377d43c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:53,686 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:53:53,686 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:53,686 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=10, startTime=1679622833625; duration=0sec 2023-03-24 01:53:53,686 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:53,688 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73322 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:53:53,689 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:53:53,689 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,689 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/492ae215a4cc487f992658d16824ee24, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=71.6 K 2023-03-24 01:53:53,689 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 492ae215a4cc487f992658d16824ee24, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1679622828714 2023-03-24 01:53:53,690 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting bb3d25f5275548738940515ab2504909, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679622828802 2023-03-24 01:53:53,690 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting cb45e44cc0f6493eae54b34055cedcfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=596, earliestPutTs=1679622829194 2023-03-24 01:53:53,691 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4da25771a15e47d380363fd998446286, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679622830398 2023-03-24 01:53:53,691 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 25974c6b232d4e51bddf4093b644234e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=642, earliestPutTs=1679622831530 2023-03-24 01:53:53,692 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f00bee47d43f4328b5ab9b5f1cabcca0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:53,707 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:53,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:53,712 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#38 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:53,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:53,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:53,720 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:53,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:53,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:53,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:53,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,737 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/305d04c44a394799b5941c45fd301bce as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/305d04c44a394799b5941c45fd301bce 2023-03-24 01:53:53,751 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 305d04c44a394799b5941c45fd301bce(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:53,751 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:53,751 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=10, startTime=1679622833625; duration=0sec 2023-03-24 01:53:53,751 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:53,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:53,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:53,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,803 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:53,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:53,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:53,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:53,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:53,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:53,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:53,809 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,810 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:53,810 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:53,810 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:53,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622833804, max timestamp=1679622833811Num uniques 859; 2023-03-24 01:53:53,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622833804, max timestamp=1679622833811Num uniques 859; 2023-03-24 01:53:53,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:53,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:53,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/4bcbb8ace97e44a795a0156d4ced9850 2023-03-24 01:53:53,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622833804, max timestamp=1679622833811Num uniques 859; 2023-03-24 01:53:53,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:53:53,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622893831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622893832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622893833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:53,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2d181f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:53,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:53,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,894 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:53,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:53,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622893897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622893905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,933 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622893936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:53,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:53,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,939 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622893936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:53,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:53,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622893937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:53,940 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:53,978 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,979 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:53,979 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:53,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:53,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:53,981 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,984 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,032 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,032 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,032 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:54,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,035 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,035 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:54,035 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:54,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:54,032 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:54,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:54,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:54,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:54,140 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:54,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,141 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622894141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622894142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622894144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:54,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:54,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:54,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:54,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:54,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:54,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:54,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:54,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4596bee8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:54,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/aaab2e43ae494a879288f530830d0578 2023-03-24 01:53:54,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:54,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,241 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:54,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,264 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:54,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:54,267 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:54,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:54,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:54,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:54,267 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:54,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,300 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:54,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:54,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:54,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,357 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:54,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,357 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:54,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622894444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622894444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622894450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,458 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:54,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:54,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:54,464 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:54,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:54,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:54,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:54,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:54,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:54,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:54,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:54,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:54,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:54,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:54,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:54,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530dd94d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:54,475 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:54,475 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:54,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:54,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:54,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:54,483 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:54,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:54,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:54,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:54,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:54,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,489 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,500 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:54,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:54,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,501 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:54,501 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:54,502 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,601 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:54,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:54,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:54,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:54,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:54,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,607 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:54,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:54,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:54,616 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,616 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:54,616 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:54,616 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:54,619 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:54,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5041648c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:54,622 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:54,622 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:54,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:54,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:54,625 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,625 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,625 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:54,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:54,630 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:54,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:54,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:54,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:54,630 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:54,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,634 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:54,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:54,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,639 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:54,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:54,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:54,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:54,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2032efa0aeef49bfaff52d123aff4510 2023-03-24 01:53:54,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/4bcbb8ace97e44a795a0156d4ced9850 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850 2023-03-24 01:53:54,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850, entries=200, sequenceid=727, filesize=14.3 K 2023-03-24 01:53:54,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/aaab2e43ae494a879288f530830d0578 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578 2023-03-24 01:53:54,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578, entries=100, sequenceid=727, filesize=9.5 K 2023-03-24 01:53:54,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2032efa0aeef49bfaff52d123aff4510 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510 2023-03-24 01:53:54,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510, entries=100, sequenceid=727, filesize=9.5 K 2023-03-24 01:53:54,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 63c1038f18949b69fb052a65a9d44cbf in 886ms, sequenceid=727, compaction requested=false 2023-03-24 01:53:54,678 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:54,740 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:54,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:54,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:54,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:54,749 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:54,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:54,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:54,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,754 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:54,754 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:54,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:54,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:54,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:54,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:54,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:54,760 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:54,760 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:54,760 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:54,760 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:54,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:54,800 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/d0234449b8b34acbb19d22aea008625b 2023-03-24 01:53:54,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:54,911 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:54,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622894933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622894933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622894946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622894949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:54,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622894958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622895037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622895037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,230 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/ead801aa62184210a9376798e7029912 2023-03-24 01:53:55,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622895240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622895240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,255 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/10baae9f7b4e4aef9c98583bc93ee302 2023-03-24 01:53:55,262 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/d0234449b8b34acbb19d22aea008625b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b 2023-03-24 01:53:55,268 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b, entries=150, sequenceid=764, filesize=11.9 K 2023-03-24 01:53:55,269 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/ead801aa62184210a9376798e7029912 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912 2023-03-24 01:53:55,277 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912, entries=150, sequenceid=764, filesize=11.9 K 2023-03-24 01:53:55,278 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/10baae9f7b4e4aef9c98583bc93ee302 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302 2023-03-24 01:53:55,284 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302, entries=150, sequenceid=764, filesize=11.9 K 2023-03-24 01:53:55,285 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 63c1038f18949b69fb052a65a9d44cbf in 525ms, sequenceid=764, compaction requested=true 2023-03-24 01:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:55,285 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:55,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:55,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:55,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14e8637b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:55,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:55,292 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:55,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:55,296 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:55,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:55,299 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:55,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:55,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:55,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:55,299 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:55,301 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,301 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,301 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,304 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:55,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:55,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:55,310 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:55,310 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:55,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:55,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:55,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:55,414 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:55,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:55,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:55,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:55,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,420 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:55,420 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:55,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,442 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/cd28c5d44a994b1d9ed2e491bb55af35 2023-03-24 01:53:55,457 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/97da0af5b4e648b8a7da69033019de8f 2023-03-24 01:53:55,476 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/3f75ced0c49f4e5899cf201ac7d4afc4 2023-03-24 01:53:55,484 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/cd28c5d44a994b1d9ed2e491bb55af35 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35 2023-03-24 01:53:55,491 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35, entries=150, sequenceid=774, filesize=11.9 K 2023-03-24 01:53:55,492 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/97da0af5b4e648b8a7da69033019de8f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f 2023-03-24 01:53:55,497 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f, entries=150, sequenceid=774, filesize=11.9 K 2023-03-24 01:53:55,498 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/3f75ced0c49f4e5899cf201ac7d4afc4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4 2023-03-24 01:53:55,503 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4, entries=150, sequenceid=774, filesize=11.9 K 2023-03-24 01:53:55,504 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 63c1038f18949b69fb052a65a9d44cbf in 84ms, sequenceid=774, compaction requested=true 2023-03-24 01:53:55,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:55,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:55,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:55,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,507 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,508 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6479fef0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:55,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:55,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:55,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:55,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:55,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:55,513 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:55,516 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:55,516 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:55,516 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:55,516 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:55,516 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:55,518 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,518 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,518 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,519 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,521 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,524 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,525 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,525 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:55,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:55,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:55,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:55,554 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:55,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622835579, max timestamp=1679622835590Num uniques 859; 2023-03-24 01:53:55,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:55,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622835579, max timestamp=1679622835590Num uniques 859; 2023-03-24 01:53:55,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622835579, max timestamp=1679622835590Num uniques 859; 2023-03-24 01:53:55,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:55,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:55,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622895612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622895613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,625 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:55,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:55,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:55,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:55,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:55,628 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:55,628 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,628 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:55,628 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:55,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,630 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:55,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:55,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:55,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:55,632 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,632 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:55,632 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:55,632 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:55,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:55,634 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,634 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59839483[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:55,653 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:55,653 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,655 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:55,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:55,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:55,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:55,660 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:55,660 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:55,660 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:55,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:55,661 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:55,663 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,663 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,663 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,666 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,668 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,668 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,668 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,668 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:55,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:55,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,668 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:55,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622895715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622895715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:55,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:55,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:55,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:55,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:55,773 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:55,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:55,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:55,774 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,774 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,776 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:55,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:55,778 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,778 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:55,778 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:55,778 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:55,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bdee3b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:55,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:55,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,784 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:55,784 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,784 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:55,786 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:55,789 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:55,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:55,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:55,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:55,789 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:55,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:55,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,797 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:55,797 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:55,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:55,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:55,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:55,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:55,900 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:55,900 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,900 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:55,900 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:55,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,902 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,902 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:55,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:55,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:55,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:55,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:55,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:55,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:55,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:55,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,911 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:55,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5519b6ac[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:55,913 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:55,913 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,914 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:55,914 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,914 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:55,916 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:55,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:55,919 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:55,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:55,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:55,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:55,919 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:55,921 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,921 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,921 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:55,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:55,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,923 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:55,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622895921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622895920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:55,930 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:55,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:55,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:55,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:55,931 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:55,931 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:55,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622895954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622895960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:55,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/fee566f8da7648fa858383877f913209 2023-03-24 01:53:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622895968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:55,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/5279eb37e4f140548c2f0da2017e27cb 2023-03-24 01:53:56,031 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:56,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:56,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:56,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:56,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:56,036 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:56,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:56,040 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,040 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:56,040 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:56,040 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:56,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebc5c81[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:56,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:56,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,047 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:56,047 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,047 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:56,048 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:56,052 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:56,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:56,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:56,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:56,052 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:56,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:56,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:56,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:56,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,059 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:56,059 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:56,159 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:56,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:56,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:56,163 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:56,163 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,163 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:56,163 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:56,164 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,164 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:56,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:56,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:56,172 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,172 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:56,172 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:56,172 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:56,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:56,174 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5ee859[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:56,176 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:56,176 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,177 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:56,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:56,183 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:56,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:56,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:56,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:56,183 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:56,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,189 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:56,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:56,192 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,192 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:56,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:56,192 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:56,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622896226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622896226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,292 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:56,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:56,296 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:56,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:56,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:56,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,298 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:56,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:56,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:56,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:56,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ff153f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:56,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:56,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,310 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:56,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:56,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:56,312 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:56,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:56,314 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:56,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:56,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:56,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:56,314 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:56,317 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,317 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,317 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,318 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:56,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,324 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:56,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,324 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:56,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=784 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/c1e88b492f6e4a46a6de279ffb539f48 2023-03-24 01:53:56,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/fee566f8da7648fa858383877f913209 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209 2023-03-24 01:53:56,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209, entries=200, sequenceid=784, filesize=14.3 K 2023-03-24 01:53:56,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/5279eb37e4f140548c2f0da2017e27cb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb 2023-03-24 01:53:56,424 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:56,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:56,429 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:56,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:56,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:56,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb, entries=150, sequenceid=784, filesize=11.9 K 2023-03-24 01:53:56,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/c1e88b492f6e4a46a6de279ffb539f48 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48 2023-03-24 01:53:56,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:56,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:56,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:56,434 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,434 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:56,435 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:56,435 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:56,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:56,436 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,438 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:56,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48, entries=150, sequenceid=784, filesize=11.9 K 2023-03-24 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e4b659a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:56,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:56,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 63c1038f18949b69fb052a65a9d44cbf in 886ms, sequenceid=784, compaction requested=true 2023-03-24 01:53:56,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:56,439 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:56,439 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:53:56,439 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:56,439 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:53:56,439 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:53:56,441 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:56,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,443 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,445 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58896 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:53:56,445 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:53:56,446 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,446 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d7e72572f54649c39f98d78ec377d43c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=57.5 K 2023-03-24 01:53:56,446 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66216 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:53:56,446 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:53:56,446 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,447 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting d7e72572f54649c39f98d78ec377d43c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:56,446 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/90b20b0f11ba40269ee885747487c413, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=64.7 K 2023-03-24 01:53:56,448 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting aaab2e43ae494a879288f530830d0578, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1679622833178 2023-03-24 01:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:56,449 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 90b20b0f11ba40269ee885747487c413, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:56,449 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:56,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:56,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:56,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:56,449 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:56,449 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4bcbb8ace97e44a795a0156d4ced9850, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1679622833135 2023-03-24 01:53:56,449 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ead801aa62184210a9376798e7029912, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1679622833830 2023-03-24 01:53:56,450 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d0234449b8b34acbb19d22aea008625b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1679622833830 2023-03-24 01:53:56,451 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting cd28c5d44a994b1d9ed2e491bb55af35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1679622834912 2023-03-24 01:53:56,451 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 97da0af5b4e648b8a7da69033019de8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1679622834912 2023-03-24 01:53:56,451 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting fee566f8da7648fa858383877f913209, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835549 2023-03-24 01:53:56,451 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5279eb37e4f140548c2f0da2017e27cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835550 2023-03-24 01:53:56,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:56,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:56,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:56,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:56,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:56,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:56,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,459 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:56,459 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:56,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:56,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:56,466 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:56,467 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#52 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:56,559 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:56,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:56,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:56,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:56,563 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:56,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:56,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:56,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,566 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:56,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:56,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:56,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:56,572 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:56,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:56,582 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/253e6b4caff64131810e553316570710 2023-03-24 01:53:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:56,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:56,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622896740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622896740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622896845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:56,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622896845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:56,892 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/5f67ccabdb0f4096a33ec05a1b8c5a3f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5f67ccabdb0f4096a33ec05a1b8c5a3f 2023-03-24 01:53:56,900 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/f7a7c85d884a4758ad344d4bf7bb801b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f7a7c85d884a4758ad344d4bf7bb801b 2023-03-24 01:53:56,907 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into 5f67ccabdb0f4096a33ec05a1b8c5a3f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:56,907 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:56,907 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=11, startTime=1679622836439; duration=0sec 2023-03-24 01:53:56,907 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:56,907 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:53:56,908 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into f7a7c85d884a4758ad344d4bf7bb801b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:56,908 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:56,908 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=11, startTime=1679622836439; duration=0sec 2023-03-24 01:53:56,909 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:56,917 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58896 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:53:56,917 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:53:56,917 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:56,917 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/305d04c44a394799b5941c45fd301bce, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=57.5 K 2023-03-24 01:53:56,918 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 305d04c44a394799b5941c45fd301bce, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1679622832798 2023-03-24 01:53:56,918 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2032efa0aeef49bfaff52d123aff4510, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1679622833178 2023-03-24 01:53:56,919 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 10baae9f7b4e4aef9c98583bc93ee302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1679622833830 2023-03-24 01:53:56,919 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3f75ced0c49f4e5899cf201ac7d4afc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1679622834912 2023-03-24 01:53:56,920 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c1e88b492f6e4a46a6de279ffb539f48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835550 2023-03-24 01:53:56,938 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#55 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:56,958 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2a48bad59a094aeea4285bd391f25a58 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a48bad59a094aeea4285bd391f25a58 2023-03-24 01:53:56,979 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 2a48bad59a094aeea4285bd391f25a58(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:56,979 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:56,979 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=11, startTime=1679622836439; duration=0sec 2023-03-24 01:53:56,979 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:57,001 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/0044449650df4aa7b3714930967f8c55 2023-03-24 01:53:57,026 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/454e389b0a6949698b0cf4420f2b3852 2023-03-24 01:53:57,033 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/253e6b4caff64131810e553316570710 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710 2023-03-24 01:53:57,040 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710, entries=150, sequenceid=819, filesize=11.9 K 2023-03-24 01:53:57,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/0044449650df4aa7b3714930967f8c55 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55 2023-03-24 01:53:57,047 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55, entries=150, sequenceid=819, filesize=11.9 K 2023-03-24 01:53:57,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,048 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/454e389b0a6949698b0cf4420f2b3852 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852 2023-03-24 01:53:57,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622897048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622897049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,059 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852, entries=150, sequenceid=819, filesize=11.9 K 2023-03-24 01:53:57,063 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 63c1038f18949b69fb052a65a9d44cbf in 491ms, sequenceid=819, compaction requested=false 2023-03-24 01:53:57,063 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:57,063 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@520f6086[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,080 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,080 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,086 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,091 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,091 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,191 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,196 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,202 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,212 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:57,212 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,213 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:57,213 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,222 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/c03d4fc3f16444ed8c4ce6a682dbe6c9 2023-03-24 01:53:57,234 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/446f3b7dfbf7496da451ad961486541f 2023-03-24 01:53:57,246 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/062c157c29fe417f8bcd52b3f2bad130 2023-03-24 01:53:57,252 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/c03d4fc3f16444ed8c4ce6a682dbe6c9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9 2023-03-24 01:53:57,258 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9, entries=150, sequenceid=832, filesize=11.9 K 2023-03-24 01:53:57,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/446f3b7dfbf7496da451ad961486541f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f 2023-03-24 01:53:57,265 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f, entries=150, sequenceid=832, filesize=11.9 K 2023-03-24 01:53:57,266 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/062c157c29fe417f8bcd52b3f2bad130 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130 2023-03-24 01:53:57,273 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130, entries=150, sequenceid=832, filesize=11.9 K 2023-03-24 01:53:57,274 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 63c1038f18949b69fb052a65a9d44cbf in 62ms, sequenceid=832, compaction requested=true 2023-03-24 01:53:57,274 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:57,274 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1edf56be[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,279 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,280 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,281 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,281 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,281 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,281 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,286 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,289 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,289 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,292 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,292 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,292 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,300 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:57,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:53:57,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:57,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:57,363 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622837375, max timestamp=1679622837393Num uniques 859; 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622837375, max timestamp=1679622837393Num uniques 859; 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622837375, max timestamp=1679622837393Num uniques 859; 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:57,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:53:57,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,404 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,405 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,405 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,406 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:57,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,412 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3027feb7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,415 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,420 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,420 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,420 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,420 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,420 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622897418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,422 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,422 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622897420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,432 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,432 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:57,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622897521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622897526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,533 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,536 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,537 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,538 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:57,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,542 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25552a9c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,544 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,544 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,550 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,550 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,552 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,552 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,552 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,552 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,556 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,557 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,557 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,557 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,557 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:57,657 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,660 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,663 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,665 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,665 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,665 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:57,665 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,667 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,667 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7775e0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,670 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,673 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,673 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,676 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,676 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,676 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,677 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,678 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622897726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622897731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/068edd102fa546bab547286cd6cfc457 2023-03-24 01:53:57,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,789 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:57,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,801 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6879b876[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,805 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,811 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,811 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,822 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,822 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,823 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,823 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,923 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:57,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:57,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:57,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:57,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:57,929 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:57,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,930 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,931 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,932 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,932 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:57,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:57,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:57,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:57,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:57,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:57,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:57,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:57,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:57,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:57,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a42188[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:57,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:57,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,948 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:57,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:57,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:57,950 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:57,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:57,953 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:57,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:57,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:57,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:57,953 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:57,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:57,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:57,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:57,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:57,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,960 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:57,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:57,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:57,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:57,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:57,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622897965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,968 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:57,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622897974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622897975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:57,978 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:57,980 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:53:58,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622898032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622898034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:58,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:58,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:58,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:58,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:58,064 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:58,064 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:58,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:58,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:58,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:58,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:58,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:58,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:58,069 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,069 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:58,069 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:58,069 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:58,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:58,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:58,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b29e62a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:58,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:58,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,075 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:58,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:58,076 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:58,079 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:58,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:58,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:58,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:58,079 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:58,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:58,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,089 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:58,090 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:58,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,190 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:58,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:58,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:58,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:58,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:58,194 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:58,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:58,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:58,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,197 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:58,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:58,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:58,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:58,204 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:58,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4656da[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:58,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:58,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,208 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:58,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:58,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:58,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:58,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,216 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/f2ce3dda078f4c1993e6b55e7ad43f3d 2023-03-24 01:53:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:58,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:58,219 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:58,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:58,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:58,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:58,219 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:58,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,230 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:58,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:58,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,238 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:58,238 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:58,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/5970bcb324d749929ccac032bfa5de58 2023-03-24 01:53:58,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/068edd102fa546bab547286cd6cfc457 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457 2023-03-24 01:53:58,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457, entries=200, sequenceid=843, filesize=14.3 K 2023-03-24 01:53:58,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/f2ce3dda078f4c1993e6b55e7ad43f3d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d 2023-03-24 01:53:58,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 01:53:58,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/5970bcb324d749929ccac032bfa5de58 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58 2023-03-24 01:53:58,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 01:53:58,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 63c1038f18949b69fb052a65a9d44cbf in 920ms, sequenceid=843, compaction requested=true 2023-03-24 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:58,282 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:53:58,282 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:58,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:53:58,284 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:53:58,285 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,285 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,285 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5f67ccabdb0f4096a33ec05a1b8c5a3f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=48.2 K 2023-03-24 01:53:58,285 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f7a7c85d884a4758ad344d4bf7bb801b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=50.6 K 2023-03-24 01:53:58,285 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5f67ccabdb0f4096a33ec05a1b8c5a3f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835550 2023-03-24 01:53:58,285 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f7a7c85d884a4758ad344d4bf7bb801b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835550 2023-03-24 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0044449650df4aa7b3714930967f8c55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=819, earliestPutTs=1679622835604 2023-03-24 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 253e6b4caff64131810e553316570710, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=819, earliestPutTs=1679622835604 2023-03-24 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 446f3b7dfbf7496da451ad961486541f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1679622836736 2023-03-24 01:53:58,286 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c03d4fc3f16444ed8c4ce6a682dbe6c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1679622836736 2023-03-24 01:53:58,287 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f2ce3dda078f4c1993e6b55e7ad43f3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:58,287 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 068edd102fa546bab547286cd6cfc457, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:58,297 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:58,298 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#63 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:58,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:58,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:58,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:58,342 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:58,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:58,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:58,347 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-24 01:53:58,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:58,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:58,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:58,348 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:58,348 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:58,348 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:53:58,357 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/5a7485d8fe3f4293afa949cc957c55b4 2023-03-24 01:53:58,372 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/7f8c028856b249d88cdd69c0aa644021 2023-03-24 01:53:58,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:58,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622898558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622898559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622898661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622898664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,715 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/bdbb11aafa5a4ac3bbdf4341a613e50a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/bdbb11aafa5a4ac3bbdf4341a613e50a 2023-03-24 01:53:58,719 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/b2e84d0b62ae4f109971d2ed496f45cd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/b2e84d0b62ae4f109971d2ed496f45cd 2023-03-24 01:53:58,726 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into bdbb11aafa5a4ac3bbdf4341a613e50a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:58,727 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:58,727 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=12, startTime=1679622838282; duration=0sec 2023-03-24 01:53:58,727 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:58,727 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:53:58,729 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:53:58,730 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:53:58,730 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,730 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a48bad59a094aeea4285bd391f25a58, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=48.2 K 2023-03-24 01:53:58,732 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2a48bad59a094aeea4285bd391f25a58, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=784, earliestPutTs=1679622835550 2023-03-24 01:53:58,733 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 454e389b0a6949698b0cf4420f2b3852, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=819, earliestPutTs=1679622835604 2023-03-24 01:53:58,733 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into b2e84d0b62ae4f109971d2ed496f45cd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:58,733 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:58,733 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=12, startTime=1679622838282; duration=0sec 2023-03-24 01:53:58,733 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:58,734 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 062c157c29fe417f8bcd52b3f2bad130, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1679622836736 2023-03-24 01:53:58,734 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5970bcb324d749929ccac032bfa5de58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:58,750 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:58,765 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/499ae78a53fa44c2b86b3ff67e6a9e0d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/499ae78a53fa44c2b86b3ff67e6a9e0d 2023-03-24 01:53:58,773 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 499ae78a53fa44c2b86b3ff67e6a9e0d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:58,773 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:58,773 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=12, startTime=1679622838282; duration=0sec 2023-03-24 01:53:58,773 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:58,786 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/0f4c083f63d24815a571fc6c3bfbdaad 2023-03-24 01:53:58,794 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/5a7485d8fe3f4293afa949cc957c55b4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4 2023-03-24 01:53:58,800 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4, entries=150, sequenceid=877, filesize=11.9 K 2023-03-24 01:53:58,801 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/7f8c028856b249d88cdd69c0aa644021 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021 2023-03-24 01:53:58,810 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021, entries=150, sequenceid=877, filesize=11.9 K 2023-03-24 01:53:58,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/0f4c083f63d24815a571fc6c3bfbdaad as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad 2023-03-24 01:53:58,820 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad, entries=150, sequenceid=877, filesize=11.9 K 2023-03-24 01:53:58,821 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 63c1038f18949b69fb052a65a9d44cbf in 474ms, sequenceid=877, compaction requested=false 2023-03-24 01:53:58,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:58,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:58,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:58,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7959ef2c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:58,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:58,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:58,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:58,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:58,832 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:58,836 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:58,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:58,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:58,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:58,836 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:58,837 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,838 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,838 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,838 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:58,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:58,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:58,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:58,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:58,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:53:58,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:53:58,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:53:58,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:58,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:53:58,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:58,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:53:58,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:53:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622898894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:58,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622898896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,944 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:58,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:58,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:58,948 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:58,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:58,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:58,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,950 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:58,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:58,952 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:58,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:58,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:58,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb739ee[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:58,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:58,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:58,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:58,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:58,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:58,965 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:58,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:58,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:58,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:58,965 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:58,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:58,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:58,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:58,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:58,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:58,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:58,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:58,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:58,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:58,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:58,975 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:58,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:58,975 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:58,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:58,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622898997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622898999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,075 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,078 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:59,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:59,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:59,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:59,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:59,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:59,089 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,090 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:59,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0b8084[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:59,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:59,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:59,095 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:59,099 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:59,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:59,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:59,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:59,099 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:59,102 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,102 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,102 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:59,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:59,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,106 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:59,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:59,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622899201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622899202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,207 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,223 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,224 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,224 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,225 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:59,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:59,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:59,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:59,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:59,230 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7595f5b9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:59,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:59,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:59,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:59,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:59,249 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:59,249 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:59,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:59,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:59,250 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:59,252 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,252 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,252 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,261 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:59,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,263 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,263 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:59,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/45ce93cbded54d7387793be74cd75117 2023-03-24 01:53:59,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/aec28782f2c54758bf9d73263bd3498f 2023-03-24 01:53:59,363 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,367 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,372 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,372 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:59,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:59,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:59,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:59,376 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:59,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:59,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:59,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:59,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:59,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:59,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,380 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,381 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438354a8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:59,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:59,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,386 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,387 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,387 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:59,389 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:59,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:59,393 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:59,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:59,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:59,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:59,393 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:59,396 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,396 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,396 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,402 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:59,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:59,402 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,402 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,402 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:59,402 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:59,402 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,402 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,502 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,505 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622899505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:53:59,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622899506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:59,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:59,510 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,510 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:59,510 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:59,510 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:59,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:59,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f9ba8b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:59,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,518 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:59,518 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,518 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:59,518 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:59,520 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:59,523 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:59,523 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:59,523 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:59,523 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:59,523 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:59,525 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,525 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,525 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,525 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,528 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:59,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:59,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:59,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:59,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,636 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:53:59,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:53:59,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:53:59,671 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,671 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:53:59,671 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:53:59,671 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:53:59,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:53:59,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2161bb1138cd4c2e96af37c034c2df91 2023-03-24 01:53:59,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/45ce93cbded54d7387793be74cd75117 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117 2023-03-24 01:53:59,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 01:53:59,731 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/aec28782f2c54758bf9d73263bd3498f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f 2023-03-24 01:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fd6d7f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:53:59,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:53:59,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 01:53:59,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2161bb1138cd4c2e96af37c034c2df91 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91 2023-03-24 01:53:59,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 01:53:59,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 63c1038f18949b69fb052a65a9d44cbf in 884ms, sequenceid=892, compaction requested=true 2023-03-24 01:53:59,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:59,749 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:59,749 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:53:59,749 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:53:59,751 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:53:59,751 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:53:59,751 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,751 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/bdbb11aafa5a4ac3bbdf4341a613e50a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.4 K 2023-03-24 01:53:59,752 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:53:59,752 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:53:59,752 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting bdbb11aafa5a4ac3bbdf4341a613e50a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:59,752 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:59,752 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/b2e84d0b62ae4f109971d2ed496f45cd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.4 K 2023-03-24 01:53:59,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:59,753 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5a7485d8fe3f4293afa949cc957c55b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1679622837416 2023-03-24 01:53:59,753 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b2e84d0b62ae4f109971d2ed496f45cd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:59,754 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 45ce93cbded54d7387793be74cd75117, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:53:59,754 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7f8c028856b249d88cdd69c0aa644021, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1679622837416 2023-03-24 01:53:59,755 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting aec28782f2c54758bf9d73263bd3498f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:53:59,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,770 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,770 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:53:59,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:53:59,784 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#72 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:59,785 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:53:59,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:53:59,788 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:53:59,788 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:53:59,788 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:53:59,788 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:53:59,788 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:53:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:53:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,792 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:53:59,793 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:59,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/8b539e17d2aa4404a2c5646c37b6370c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/8b539e17d2aa4404a2c5646c37b6370c 2023-03-24 01:53:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:53:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,824 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:53:59,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,832 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/996567cc197a4371b786428a55a587d6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/996567cc197a4371b786428a55a587d6 2023-03-24 01:53:59,832 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into 8b539e17d2aa4404a2c5646c37b6370c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:59,832 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:59,832 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=13, startTime=1679622839749; duration=0sec 2023-03-24 01:53:59,832 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:53:59,832 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:53:59,838 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:53:59,838 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:53:59,838 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:53:59,841 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/499ae78a53fa44c2b86b3ff67e6a9e0d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.4 K 2023-03-24 01:53:59,842 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 499ae78a53fa44c2b86b3ff67e6a9e0d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679622837357 2023-03-24 01:53:59,843 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0f4c083f63d24815a571fc6c3bfbdaad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1679622837416 2023-03-24 01:53:59,845 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2161bb1138cd4c2e96af37c034c2df91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:53:59,848 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into 996567cc197a4371b786428a55a587d6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:53:59,848 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:53:59,848 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=13, startTime=1679622839749; duration=0sec 2023-03-24 01:53:59,849 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:53:59,854 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#74 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:53:59,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:53:59,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:53:59,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:53:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:53:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:53:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:53:59,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:53:59,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:53:59,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:53:59,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:53:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:53:59,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:53:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:53:59,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:53:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:53:59,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:53:59,977 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:53:59,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:53:59,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:53:59,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:53:59,978 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:53:59,978 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,008 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,010 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,010 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:00,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622838887, max timestamp=1679622840011Num uniques 859; 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622838887, max timestamp=1679622840011Num uniques 859; 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622838887, max timestamp=1679622840011Num uniques 859; 2023-03-24 01:54:00,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:00,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:00,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:00,016 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,016 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,016 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,016 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,016 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622840037, max timestamp=1679622840049Num uniques 859; 2023-03-24 01:54:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622840037, max timestamp=1679622840049Num uniques 859; 2023-03-24 01:54:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622840037, max timestamp=1679622840049Num uniques 859; 2023-03-24 01:54:00,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:00,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5cae4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622840069, max timestamp=1679622840093Num uniques 1018; 2023-03-24 01:54:00,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:54:00,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622840069, max timestamp=1679622840093Num uniques 1018; 2023-03-24 01:54:00,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:54:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622840069, max timestamp=1679622840093Num uniques 1018; 2023-03-24 01:54:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:54:00,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,139 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679622840137, max timestamp=1679622840158Num uniques 1127; 2023-03-24 01:54:00,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679622840137, max timestamp=1679622840158Num uniques 1127; 2023-03-24 01:54:00,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-24 01:54:00,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-24 01:54:00,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679622840137, max timestamp=1679622840158Num uniques 1127; 2023-03-24 01:54:00,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-03-24 01:54:00,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622900209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622900214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622840202, max timestamp=1679622840212Num uniques 1136; 2023-03-24 01:54:00,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 01:54:00,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622840202, max timestamp=1679622840212Num uniques 1136; 2023-03-24 01:54:00,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 01:54:00,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679622840202, max timestamp=1679622840212Num uniques 1136; 2023-03-24 01:54:00,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 01:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,238 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,238 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,283 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/2a01c76c35d749cdaff88c2cac4bc8dc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a01c76c35d749cdaff88c2cac4bc8dc 2023-03-24 01:54:00,284 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,284 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,285 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,286 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,297 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 2a01c76c35d749cdaff88c2cac4bc8dc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:00,297 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:00,297 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=13, startTime=1679622839752; duration=0sec 2023-03-24 01:54:00,297 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,308 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:00,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:00,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,308 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:00,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:00,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1679622840315, max timestamp=1679622840349Num uniques 1045; 2023-03-24 01:54:00,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=13500, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-24 01:54:00,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1679622840315, max timestamp=1679622840349Num uniques 1045; 2023-03-24 01:54:00,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1679622840315, max timestamp=1679622840349Num uniques 1045; 2023-03-24 01:54:00,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=13500, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-24 01:54:00,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47850, new segment data size=13500, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-03-24 01:54:00,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679622840381, max timestamp=1679622840393Num uniques 1154; 2023-03-24 01:54:00,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-24 01:54:00,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679622840381, max timestamp=1679622840393Num uniques 1154; 2023-03-24 01:54:00,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-24 01:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1679622840381, max timestamp=1679622840393Num uniques 1154; 2023-03-24 01:54:00,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-24 01:54:00,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:00,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:00,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:00,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:00,415 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,422 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,424 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,425 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,425 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5857cf36[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/c2a60fb4959b48d8a3a69a281a566d0a 2023-03-24 01:54:00,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622840428, max timestamp=1679622840435Num uniques 1063; 2023-03-24 01:54:00,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622840428, max timestamp=1679622840435Num uniques 1063; 2023-03-24 01:54:00,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-24 01:54:00,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679622840428, max timestamp=1679622840435Num uniques 1063; 2023-03-24 01:54:00,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-24 01:54:00,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-03-24 01:54:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,442 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,445 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,446 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/e8edaabbbb7741438810c8850c827414 2023-03-24 01:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:00,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:00,457 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,457 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:00,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=916 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/6469cf5b2dbf49328ef9aadaf6fb6602 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679622840470, max timestamp=1679622840484Num uniques 1072; 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679622840470, max timestamp=1679622840484Num uniques 1072; 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1679622840470, max timestamp=1679622840484Num uniques 1072; 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-24 01:54:00,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-03-24 01:54:00,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/c2a60fb4959b48d8a3a69a281a566d0a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a 2023-03-24 01:54:00,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 01:54:00,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/e8edaabbbb7741438810c8850c827414 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414 2023-03-24 01:54:00,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 01:54:00,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/6469cf5b2dbf49328ef9aadaf6fb6602 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602 2023-03-24 01:54:00,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602, entries=150, sequenceid=916, filesize=11.9 K 2023-03-24 01:54:00,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=114.05 KB/116790 for 63c1038f18949b69fb052a65a9d44cbf in 509ms, sequenceid=916, compaction requested=false 2023-03-24 01:54:00,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:00,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=120.76 KB heapSize=325.17 KB 2023-03-24 01:54:00,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:00,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:00,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:00,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:00,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622900541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622900542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/6a1d76aac7f04e0d966831b5ce7fb795 2023-03-24 01:54:00,557 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:00,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:00,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:00,572 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:00,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:00,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,575 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,581 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/40277ffdd51c4ca297c33af6e233332b 2023-03-24 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,584 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,589 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,589 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128e680[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,600 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,600 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,602 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:00,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,612 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:00,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622900646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622900646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:00,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:00,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:00,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:00,717 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:00,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:00,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:00,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,721 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,723 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f10f54f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,726 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,726 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,733 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,733 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:00,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:00,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,740 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:00,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:00,740 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:00,840 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:00,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:00,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:00,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:00,844 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:00,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:00,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:00,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,846 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ef8a3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622900849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622900850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,854 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,859 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,860 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,862 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,862 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:00,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,869 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:00,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:00,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:00,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:00,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:00,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:00,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:00,974 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:00,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:00,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:00,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,976 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:00,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:00,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:00,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:00,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:00,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:00,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:00,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:00,980 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6516b981[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:00,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:00,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,984 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:00,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:00,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:00,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:00,988 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:00,991 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:00,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:00,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:00,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:00,992 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:00,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:00,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:00,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,001 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,006 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:01,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:01,007 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:01,007 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1068 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/8c3deb445006441eb5508f84630134ae 2023-03-24 01:54:01,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/6a1d76aac7f04e0d966831b5ce7fb795 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795 2023-03-24 01:54:01,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795, entries=300, sequenceid=1068, filesize=19.0 K 2023-03-24 01:54:01,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/40277ffdd51c4ca297c33af6e233332b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b 2023-03-24 01:54:01,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b, entries=150, sequenceid=1068, filesize=11.9 K 2023-03-24 01:54:01,029 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/8c3deb445006441eb5508f84630134ae as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae 2023-03-24 01:54:01,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae, entries=150, sequenceid=1068, filesize=11.9 K 2023-03-24 01:54:01,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~325.13 KB/332928, currentSize=67.09 KB/68700 for 63c1038f18949b69fb052a65a9d44cbf in 515ms, sequenceid=1068, compaction requested=true 2023-03-24 01:54:01,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:01,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:01,038 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:01,038 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:01,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:01,038 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:01,039 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:01,040 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:54:01,040 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,040 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:01,040 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/8b539e17d2aa4404a2c5646c37b6370c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=43.6 K 2023-03-24 01:54:01,040 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:54:01,040 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,040 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/996567cc197a4371b786428a55a587d6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.5 K 2023-03-24 01:54:01,040 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8b539e17d2aa4404a2c5646c37b6370c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:54:01,041 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 996567cc197a4371b786428a55a587d6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:54:01,041 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c2a60fb4959b48d8a3a69a281a566d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679622838894 2023-03-24 01:54:01,041 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e8edaabbbb7741438810c8850c827414, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679622838894 2023-03-24 01:54:01,042 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6a1d76aac7f04e0d966831b5ce7fb795, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:01,042 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 40277ffdd51c4ca297c33af6e233332b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:01,053 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:01,057 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:01,107 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:01,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:01,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:01,111 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:01,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:01,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:01,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,113 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:01,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:01,128 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:01,128 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,128 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:01,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:01,145 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:01,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:01,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:01,163 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/de4bbdd2fb8a4dcc85f090925307f457 2023-03-24 01:54:01,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622901187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622901187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622901291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622901291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,477 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/431d217f4e554c08bbcacc4995ab6b59 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/431d217f4e554c08bbcacc4995ab6b59 2023-03-24 01:54:01,478 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/f5c0ae1d46d9481ea1a3aa1de9884bb3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f5c0ae1d46d9481ea1a3aa1de9884bb3 2023-03-24 01:54:01,482 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into 431d217f4e554c08bbcacc4995ab6b59(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:01,483 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:01,483 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=13, startTime=1679622841038; duration=0sec 2023-03-24 01:54:01,483 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:01,483 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:01,484 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into f5c0ae1d46d9481ea1a3aa1de9884bb3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:01,484 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:01,484 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=13, startTime=1679622841038; duration=0sec 2023-03-24 01:54:01,484 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:01,484 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:01,485 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:54:01,485 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,485 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a01c76c35d749cdaff88c2cac4bc8dc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.5 K 2023-03-24 01:54:01,485 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2a01c76c35d749cdaff88c2cac4bc8dc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1679622838555 2023-03-24 01:54:01,486 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6469cf5b2dbf49328ef9aadaf6fb6602, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=916, earliestPutTs=1679622838894 2023-03-24 01:54:01,486 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8c3deb445006441eb5508f84630134ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:01,492 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:01,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622901494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622901495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,579 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/abc1df7d95f046d0b2761f58d391fe4d 2023-03-24 01:54:01,594 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/98bcb6b2b82241ec8fb5cb059df80e9b 2023-03-24 01:54:01,607 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/de4bbdd2fb8a4dcc85f090925307f457 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457 2023-03-24 01:54:01,619 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457, entries=150, sequenceid=1081, filesize=11.9 K 2023-03-24 01:54:01,620 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/abc1df7d95f046d0b2761f58d391fe4d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d 2023-03-24 01:54:01,640 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d, entries=150, sequenceid=1081, filesize=11.9 K 2023-03-24 01:54:01,645 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/98bcb6b2b82241ec8fb5cb059df80e9b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b 2023-03-24 01:54:01,652 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b, entries=150, sequenceid=1081, filesize=11.9 K 2023-03-24 01:54:01,675 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 63c1038f18949b69fb052a65a9d44cbf in 530ms, sequenceid=1081, compaction requested=false 2023-03-24 01:54:01,675 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:01,675 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:01,675 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:01,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23cc5f45[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:01,680 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:01,680 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,682 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:01,682 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:01,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:01,685 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:01,689 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:01,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:01,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:01,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:01,689 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:01,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:01,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,697 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:01,697 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:01,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,797 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:01,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:01,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:54:01,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:01,798 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:01,798 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,798 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:01,798 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,798 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:01,799 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:01,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:01,801 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:01,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:01,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:01,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,805 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:01,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:01,809 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,809 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:01,809 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:01,809 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:01,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:01,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bde6cfe[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:01,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:01,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622901813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622901814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:01,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:01,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:01,820 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:01,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:01,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:01,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:01,820 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:01,822 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,822 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,822 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:01,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:01,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:01,831 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:01,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,908 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/78125248c0de4515b42d1f4512321650 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/78125248c0de4515b42d1f4512321650 2023-03-24 01:54:01,914 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 78125248c0de4515b42d1f4512321650(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:01,914 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:01,914 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=13, startTime=1679622841038; duration=0sec 2023-03-24 01:54:01,914 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:01,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622901916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622901917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,931 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:01,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:01,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:01,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:01,934 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:01,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,936 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:01,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:01,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:01,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:01,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:01,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:01,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:01,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:01,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:01,944 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,944 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,945 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:01,945 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3106c6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:01,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:01,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,948 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:01,949 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,949 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:01,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:01,956 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:01,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:01,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:01,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:01,956 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:01,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:01,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:01,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,960 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:01,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:01,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:01,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:01,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:01,962 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:01,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:01,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:01,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:01,963 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:01,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:01,963 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:01,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:01,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48774 deadline: 1679622901991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,994 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:01,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:01,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48794 deadline: 1679622901997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:01,999 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:02,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48754 deadline: 1679622902004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,007 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:02,063 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,066 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:02,073 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,073 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbb7210[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:02,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:02,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,076 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:02,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:02,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:02,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:02,082 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:02,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:02,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:02,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:02,082 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:02,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,086 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:02,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:02,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,090 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:02,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,090 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:02,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622902119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622902122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,191 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,194 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:02,199 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:02,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ecc26d5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:02,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:02,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:02,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:02,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:02,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:02,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:02,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/be1a1fab64324e3885785000e2939959 2023-03-24 01:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:02,209 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:02,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:02,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:02,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:02,209 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:02,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:02,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:02,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:02,221 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,222 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:02,222 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:02,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/08d1ce499f654eee8173fc0f1da017a8 2023-03-24 01:54:02,322 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,325 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,333 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:02,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:02,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:02,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:02,334 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:02,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160d1e6e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:02,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:02,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,338 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:02,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:02,342 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:02,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:02,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:02,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:02,343 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:02,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:02,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:02,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:02,351 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:02,351 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:02,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622902424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622902425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,451 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,454 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:02,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:02,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@617d3d5f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:02,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:02,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:02,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:02,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:02,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:02,465 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:02,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:02,467 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:02,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:02,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:02,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:02,467 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:02,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,473 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,475 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:02,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,478 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:02,478 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:02,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,578 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,600 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,600 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,600 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,600 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:02,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1522f892[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:02,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:02,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:02,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:02,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:02,610 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:02,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:02,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:02,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:02,611 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:02,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:02,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:02,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,614 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:02,617 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:02,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:02,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:02,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:02,617 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:02,617 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:02,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:02,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/d4f30c6ac18e44c9b1ca344d4945a05b 2023-03-24 01:54:02,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/be1a1fab64324e3885785000e2939959 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959 2023-03-24 01:54:02,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959, entries=200, sequenceid=1103, filesize=14.3 K 2023-03-24 01:54:02,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/08d1ce499f654eee8173fc0f1da017a8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8 2023-03-24 01:54:02,703 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8, entries=150, sequenceid=1103, filesize=11.9 K 2023-03-24 01:54:02,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/d4f30c6ac18e44c9b1ca344d4945a05b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b 2023-03-24 01:54:02,717 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:02,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:02,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:02,722 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,726 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:02,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:02,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b, entries=150, sequenceid=1103, filesize=11.9 K 2023-03-24 01:54:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,729 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:02,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:02,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 63c1038f18949b69fb052a65a9d44cbf in 931ms, sequenceid=1103, compaction requested=true 2023-03-24 01:54:02,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:02,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:02,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:02,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:02,729 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:02,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:02,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:54:02,729 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:02,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:02,732 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:02,732 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:54:02,732 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,732 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f5c0ae1d46d9481ea1a3aa1de9884bb3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.6 K 2023-03-24 01:54:02,732 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:02,732 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:54:02,732 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,732 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/431d217f4e554c08bbcacc4995ab6b59, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=39.0 K 2023-03-24 01:54:02,733 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f5c0ae1d46d9481ea1a3aa1de9884bb3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:02,734 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting abc1df7d95f046d0b2761f58d391fe4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679622840526 2023-03-24 01:54:02,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:02,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:02,734 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 431d217f4e554c08bbcacc4995ab6b59, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:02,735 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting de4bbdd2fb8a4dcc85f090925307f457, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679622840526 2023-03-24 01:54:02,735 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting be1a1fab64324e3885785000e2939959, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:02,735 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 08d1ce499f654eee8173fc0f1da017a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:02,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. started... 2023-03-24 01:54:02,736 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:54:02,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:02,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:02,736 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:02,737 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:02,737 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:02,737 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:02,761 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:02,770 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:02,784 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1116 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/ad9acba350424a24b9294bc765b11947 2023-03-24 01:54:02,792 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/0e1d253d67104673a380351be85faeeb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/0e1d253d67104673a380351be85faeeb 2023-03-24 01:54:02,799 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into 0e1d253d67104673a380351be85faeeb(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:02,799 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:02,799 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=13, startTime=1679622842729; duration=0sec 2023-03-24 01:54:02,799 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:02,799 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:02,803 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:02,803 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:54:02,803 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:02,803 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/78125248c0de4515b42d1f4512321650, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.6 K 2023-03-24 01:54:02,804 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 78125248c0de4515b42d1f4512321650, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1068, earliestPutTs=1679622840484 2023-03-24 01:54:02,805 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 98bcb6b2b82241ec8fb5cb059df80e9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679622840526 2023-03-24 01:54:02,805 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d4f30c6ac18e44c9b1ca344d4945a05b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:02,808 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/c410d25b4a2f4865b4b57920ccc099fd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c410d25b4a2f4865b4b57920ccc099fd 2023-03-24 01:54:02,817 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:02,817 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into c410d25b4a2f4865b4b57920ccc099fd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:02,818 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:02,818 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=13, startTime=1679622842729; duration=0sec 2023-03-24 01:54:02,818 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:02,831 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/9330baec5dcd4a7e8c903aafb6d38898 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9330baec5dcd4a7e8c903aafb6d38898 2023-03-24 01:54:02,836 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 9330baec5dcd4a7e8c903aafb6d38898(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:02,836 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:02,836 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=13, startTime=1679622842729; duration=0sec 2023-03-24 01:54:02,836 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:02,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:02,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. as already flushing 2023-03-24 01:54:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622902956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:02,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622902957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622903059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622903059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,221 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1116 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/6701ff7da47d44eab1fcd92abdba9f39 2023-03-24 01:54:03,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622903262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622903262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,476 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:62686 2023-03-24 01:54:03,476 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:03,477 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:62686 2023-03-24 01:54:03,477 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:03,477 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:62686 2023-03-24 01:54:03,478 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:03,478 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:62686 2023-03-24 01:54:03,478 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:03,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:03,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48802 deadline: 1679622903566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48746 deadline: 1679622903566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,643 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1116 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/33ee5dc740854809821acf8f7d55bfe1 2023-03-24 01:54:03,647 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/ad9acba350424a24b9294bc765b11947 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947 2023-03-24 01:54:03,651 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947, entries=150, sequenceid=1116, filesize=11.9 K 2023-03-24 01:54:03,652 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/6701ff7da47d44eab1fcd92abdba9f39 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39 2023-03-24 01:54:03,656 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39, entries=150, sequenceid=1116, filesize=11.9 K 2023-03-24 01:54:03,657 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/33ee5dc740854809821acf8f7d55bfe1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1 2023-03-24 01:54:03,662 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1, entries=150, sequenceid=1116, filesize=11.9 K 2023-03-24 01:54:03,663 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 63c1038f18949b69fb052a65a9d44cbf in 927ms, sequenceid=1116, compaction requested=false 2023-03-24 01:54:03,663 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:03,663 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:03,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,668 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a579e6d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:03,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:03,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,672 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:03,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:03,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:03,674 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:03,676 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:03,676 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:03,676 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:03,676 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:03,676 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:03,677 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,677 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,677 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:03,678 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:03,679 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,682 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,682 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:03,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:03,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:03,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:03,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:03,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:03,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:03,683 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:03,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:03,683 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:03,783 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:03,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:04,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:04,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 01:54:04,069 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:62686 2023-03-24 01:54:04,069 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:04,069 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:62686 2023-03-24 01:54:04,069 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:04,069 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:04,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1141 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/dd2850f96d894e59a93309a1ae00f4af 2023-03-24 01:54:04,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1141 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1a1f1cdb42904f9c8e1d0c9580c668d1 2023-03-24 01:54:05,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1141 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/14153078dfac4797ab3c105089a1d682 2023-03-24 01:54:05,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/dd2850f96d894e59a93309a1ae00f4af as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af 2023-03-24 01:54:05,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af, entries=150, sequenceid=1141, filesize=11.9 K 2023-03-24 01:54:05,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/1a1f1cdb42904f9c8e1d0c9580c668d1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1 2023-03-24 01:54:05,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1, entries=150, sequenceid=1141, filesize=11.9 K 2023-03-24 01:54:05,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/14153078dfac4797ab3c105089a1d682 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682 2023-03-24 01:54:05,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682, entries=150, sequenceid=1141, filesize=11.9 K 2023-03-24 01:54:05,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=0 B/0 for 63c1038f18949b69fb052a65a9d44cbf in 1251ms, sequenceid=1141, compaction requested=true 2023-03-24 01:54:05,321 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:05,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:05,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:05,321 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:05,321 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:05,321 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:05,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:05,322 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:05,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/A is initiating minor compaction (all files) 2023-03-24 01:54:05,322 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/B is initiating minor compaction (all files) 2023-03-24 01:54:05,322 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/A in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:05,322 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/B in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:05,322 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/0e1d253d67104673a380351be85faeeb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.7 K 2023-03-24 01:54:05,322 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c410d25b4a2f4865b4b57920ccc099fd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.7 K 2023-03-24 01:54:05,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0e1d253d67104673a380351be85faeeb, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:05,323 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c410d25b4a2f4865b4b57920ccc099fd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:05,323 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ad9acba350424a24b9294bc765b11947, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1116, earliestPutTs=1679622841811 2023-03-24 01:54:05,323 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6701ff7da47d44eab1fcd92abdba9f39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1116, earliestPutTs=1679622841811 2023-03-24 01:54:05,323 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting dd2850f96d894e59a93309a1ae00f4af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1141, earliestPutTs=1679622842954 2023-03-24 01:54:05,323 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1a1f1cdb42904f9c8e1d0c9580c668d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1141, earliestPutTs=1679622842954 2023-03-24 01:54:05,336 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#A#compaction#100 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:05,336 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#B#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:05,345 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/fcbf7be61ecb445fb974e7467947a145 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fcbf7be61ecb445fb974e7467947a145 2023-03-24 01:54:05,350 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/A of 63c1038f18949b69fb052a65a9d44cbf into fcbf7be61ecb445fb974e7467947a145(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:05,350 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:05,350 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/A, priority=13, startTime=1679622845321; duration=0sec 2023-03-24 01:54:05,350 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:05,350 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:05,351 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:05,351 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 63c1038f18949b69fb052a65a9d44cbf/C is initiating minor compaction (all files) 2023-03-24 01:54:05,351 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 63c1038f18949b69fb052a65a9d44cbf/C in TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:05,351 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9330baec5dcd4a7e8c903aafb6d38898, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp, totalSize=36.7 K 2023-03-24 01:54:05,351 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9330baec5dcd4a7e8c903aafb6d38898, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1679622841184 2023-03-24 01:54:05,352 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 33ee5dc740854809821acf8f7d55bfe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1116, earliestPutTs=1679622841811 2023-03-24 01:54:05,352 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 14153078dfac4797ab3c105089a1d682, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1141, earliestPutTs=1679622842954 2023-03-24 01:54:05,358 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 63c1038f18949b69fb052a65a9d44cbf#C#compaction#101 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:05,747 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/9df7018104094fdaba80b29b54d042f7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/9df7018104094fdaba80b29b54d042f7 2023-03-24 01:54:05,752 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/B of 63c1038f18949b69fb052a65a9d44cbf into 9df7018104094fdaba80b29b54d042f7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:05,752 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:05,752 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/B, priority=13, startTime=1679622845321; duration=0sec 2023-03-24 01:54:05,752 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:05,767 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/23c16efbc5bb419cbad591f0eacb1913 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/23c16efbc5bb419cbad591f0eacb1913 2023-03-24 01:54:05,772 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 63c1038f18949b69fb052a65a9d44cbf/C of 63c1038f18949b69fb052a65a9d44cbf into 23c16efbc5bb419cbad591f0eacb1913(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:05,772 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:05,772 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf., storeName=63c1038f18949b69fb052a65a9d44cbf/C, priority=13, startTime=1679622845321; duration=0sec 2023-03-24 01:54:05,772 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:12,012 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:62686 2023-03-24 01:54:12,012 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:12,061 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:62686 2023-03-24 01:54:12,061 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:12,092 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:62686 2023-03-24 01:54:12,092 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:12,092 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 287 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 303 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 3999 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4113 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1804 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5405 rows 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1777 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5318 rows 2023-03-24 01:54:12,093 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:54:12,093 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44ef43e3 to 127.0.0.1:62686 2023-03-24 01:54:12,093 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:12,097 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:54:12,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:54:12,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:12,111 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622852110"}]},"ts":"1679622852110"} 2023-03-24 01:54:12,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:12,112 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:54:12,114 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:54:12,115 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, UNASSIGN}] 2023-03-24 01:54:12,116 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, UNASSIGN 2023-03-24 01:54:12,117 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=63c1038f18949b69fb052a65a9d44cbf, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:12,117 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622852117"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622852117"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622852117"}]},"ts":"1679622852117"} 2023-03-24 01:54:12,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:12,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:12,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:12,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 63c1038f18949b69fb052a65a9d44cbf, disabling compactions & flushes 2023-03-24 01:54:12,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:12,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:12,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. after waiting 0 ms 2023-03-24 01:54:12,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:12,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 63c1038f18949b69fb052a65a9d44cbf 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=A 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=B 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 63c1038f18949b69fb052a65a9d44cbf, store=C 2023-03-24 01:54:12,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:12,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:12,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1150 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/9aa6cae8c2b3452a8707051809b22fee 2023-03-24 01:54:12,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:13,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1150 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/d8f9dfd36ed54e6d9128542a190fd2d5 2023-03-24 01:54:13,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:13,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1150 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/71956147e2fa4fc38e2bd9841a1c2ba5 2023-03-24 01:54:13,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/A/9aa6cae8c2b3452a8707051809b22fee as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9aa6cae8c2b3452a8707051809b22fee 2023-03-24 01:54:13,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9aa6cae8c2b3452a8707051809b22fee, entries=150, sequenceid=1150, filesize=11.9 K 2023-03-24 01:54:13,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/B/d8f9dfd36ed54e6d9128542a190fd2d5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d8f9dfd36ed54e6d9128542a190fd2d5 2023-03-24 01:54:13,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d8f9dfd36ed54e6d9128542a190fd2d5, entries=150, sequenceid=1150, filesize=11.9 K 2023-03-24 01:54:13,525 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/.tmp/C/71956147e2fa4fc38e2bd9841a1c2ba5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/71956147e2fa4fc38e2bd9841a1c2ba5 2023-03-24 01:54:13,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/71956147e2fa4fc38e2bd9841a1c2ba5, entries=150, sequenceid=1150, filesize=11.9 K 2023-03-24 01:54:13,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 63c1038f18949b69fb052a65a9d44cbf in 1254ms, sequenceid=1150, compaction requested=false 2023-03-24 01:54:13,549 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/51a0c624d6f24445b8bccd91b523b50a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/90b20b0f11ba40269ee885747487c413, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f7a7c85d884a4758ad344d4bf7bb801b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/bdbb11aafa5a4ac3bbdf4341a613e50a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/8b539e17d2aa4404a2c5646c37b6370c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/431d217f4e554c08bbcacc4995ab6b59, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/0e1d253d67104673a380351be85faeeb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af] to archive 2023-03-24 01:54:13,551 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:13,556 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/805f4957f7484eb086aff87b598515d8 2023-03-24 01:54:13,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/32ccd25caf1040c28cc8e30fd6441c86 2023-03-24 01:54:13,558 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/89115f5c98e247e8b9861f8ba5f98fa3 2023-03-24 01:54:13,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9a8303f09e5e4a418f8995d48bcffb2a 2023-03-24 01:54:13,561 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/2ff03e2ca0304c71933b4f6341423f00 2023-03-24 01:54:13,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/639603e6f00e42159818e91328c07be0 2023-03-24 01:54:13,563 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/51a0c624d6f24445b8bccd91b523b50a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/51a0c624d6f24445b8bccd91b523b50a 2023-03-24 01:54:13,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/7527c006a2884a469f6dfd287a4c837c 2023-03-24 01:54:13,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f13793f53e4f4923bbdd4adb7c260f3d 2023-03-24 01:54:13,566 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d008c2825d9f4428ab806ac6be0e6679 2023-03-24 01:54:13,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b8e845084eaf4d8bb52b4ec93570640e 2023-03-24 01:54:13,569 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/b527642ecbdb4ddebe68008c767c54f4 2023-03-24 01:54:13,570 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/90b20b0f11ba40269ee885747487c413 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/90b20b0f11ba40269ee885747487c413 2023-03-24 01:54:13,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/4bcbb8ace97e44a795a0156d4ced9850 2023-03-24 01:54:13,572 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/d0234449b8b34acbb19d22aea008625b 2023-03-24 01:54:13,573 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/cd28c5d44a994b1d9ed2e491bb55af35 2023-03-24 01:54:13,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fee566f8da7648fa858383877f913209 2023-03-24 01:54:13,576 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f7a7c85d884a4758ad344d4bf7bb801b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/f7a7c85d884a4758ad344d4bf7bb801b 2023-03-24 01:54:13,577 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/253e6b4caff64131810e553316570710 2023-03-24 01:54:13,578 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c03d4fc3f16444ed8c4ce6a682dbe6c9 2023-03-24 01:54:13,579 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/068edd102fa546bab547286cd6cfc457 2023-03-24 01:54:13,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/bdbb11aafa5a4ac3bbdf4341a613e50a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/bdbb11aafa5a4ac3bbdf4341a613e50a 2023-03-24 01:54:13,581 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/5a7485d8fe3f4293afa949cc957c55b4 2023-03-24 01:54:13,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/8b539e17d2aa4404a2c5646c37b6370c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/8b539e17d2aa4404a2c5646c37b6370c 2023-03-24 01:54:13,583 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/45ce93cbded54d7387793be74cd75117 2023-03-24 01:54:13,584 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/c2a60fb4959b48d8a3a69a281a566d0a 2023-03-24 01:54:13,585 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/6a1d76aac7f04e0d966831b5ce7fb795 2023-03-24 01:54:13,585 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/431d217f4e554c08bbcacc4995ab6b59 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/431d217f4e554c08bbcacc4995ab6b59 2023-03-24 01:54:13,587 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/de4bbdd2fb8a4dcc85f090925307f457 2023-03-24 01:54:13,588 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/be1a1fab64324e3885785000e2939959 2023-03-24 01:54:13,589 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/0e1d253d67104673a380351be85faeeb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/0e1d253d67104673a380351be85faeeb 2023-03-24 01:54:13,590 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/ad9acba350424a24b9294bc765b11947 2023-03-24 01:54:13,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/dd2850f96d894e59a93309a1ae00f4af 2023-03-24 01:54:13,615 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6059547a18014e24aca78e317c96c8a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d7e72572f54649c39f98d78ec377d43c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5f67ccabdb0f4096a33ec05a1b8c5a3f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/b2e84d0b62ae4f109971d2ed496f45cd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/996567cc197a4371b786428a55a587d6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f5c0ae1d46d9481ea1a3aa1de9884bb3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c410d25b4a2f4865b4b57920ccc099fd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1] to archive 2023-03-24 01:54:13,616 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:13,617 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/8c27f6bb491a42fcbe49377ce6e11a5c 2023-03-24 01:54:13,619 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1ab71d1290fe427abb03026052a2559d 2023-03-24 01:54:13,620 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/438cd393faf64ca7bc4a26f5cb09fd96 2023-03-24 01:54:13,621 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5e4d38a627ce43d1a13aed5bad215808 2023-03-24 01:54:13,622 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ac4e787dfd324e8093f4595cf3006bd3 2023-03-24 01:54:13,623 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6059547a18014e24aca78e317c96c8a2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6059547a18014e24aca78e317c96c8a2 2023-03-24 01:54:13,624 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c1e418f4582046c99da248cb0f25508e 2023-03-24 01:54:13,626 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/2c45573dd9724b12baf29630cb6d4455 2023-03-24 01:54:13,627 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/54e8dfde0b0e4c23ae272cf2fe6156b9 2023-03-24 01:54:13,628 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/3957caf72d4b4ea494993bc852a69cf7 2023-03-24 01:54:13,629 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1b75a7d9605d45ddaf400402be1ec46e 2023-03-24 01:54:13,630 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d7e72572f54649c39f98d78ec377d43c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d7e72572f54649c39f98d78ec377d43c 2023-03-24 01:54:13,631 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/db7b54ddd1ab440690c23c9188d0de32 2023-03-24 01:54:13,632 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aaab2e43ae494a879288f530830d0578 2023-03-24 01:54:13,633 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/ead801aa62184210a9376798e7029912 2023-03-24 01:54:13,634 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/97da0af5b4e648b8a7da69033019de8f 2023-03-24 01:54:13,635 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5f67ccabdb0f4096a33ec05a1b8c5a3f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5f67ccabdb0f4096a33ec05a1b8c5a3f 2023-03-24 01:54:13,637 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/5279eb37e4f140548c2f0da2017e27cb 2023-03-24 01:54:13,638 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/0044449650df4aa7b3714930967f8c55 2023-03-24 01:54:13,639 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/446f3b7dfbf7496da451ad961486541f 2023-03-24 01:54:13,640 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/b2e84d0b62ae4f109971d2ed496f45cd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/b2e84d0b62ae4f109971d2ed496f45cd 2023-03-24 01:54:13,642 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f2ce3dda078f4c1993e6b55e7ad43f3d 2023-03-24 01:54:13,643 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/7f8c028856b249d88cdd69c0aa644021 2023-03-24 01:54:13,644 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/996567cc197a4371b786428a55a587d6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/996567cc197a4371b786428a55a587d6 2023-03-24 01:54:13,646 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/aec28782f2c54758bf9d73263bd3498f 2023-03-24 01:54:13,647 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/e8edaabbbb7741438810c8850c827414 2023-03-24 01:54:13,649 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f5c0ae1d46d9481ea1a3aa1de9884bb3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/f5c0ae1d46d9481ea1a3aa1de9884bb3 2023-03-24 01:54:13,650 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/40277ffdd51c4ca297c33af6e233332b 2023-03-24 01:54:13,652 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/abc1df7d95f046d0b2761f58d391fe4d 2023-03-24 01:54:13,653 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c410d25b4a2f4865b4b57920ccc099fd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/c410d25b4a2f4865b4b57920ccc099fd 2023-03-24 01:54:13,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/08d1ce499f654eee8173fc0f1da017a8 2023-03-24 01:54:13,656 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/6701ff7da47d44eab1fcd92abdba9f39 2023-03-24 01:54:13,657 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/1a1f1cdb42904f9c8e1d0c9580c668d1 2023-03-24 01:54:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/492ae215a4cc487f992658d16824ee24, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/305d04c44a394799b5941c45fd301bce, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a48bad59a094aeea4285bd391f25a58, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/499ae78a53fa44c2b86b3ff67e6a9e0d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a01c76c35d749cdaff88c2cac4bc8dc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/78125248c0de4515b42d1f4512321650, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9330baec5dcd4a7e8c903aafb6d38898, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682] to archive 2023-03-24 01:54:13,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:13,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/e445c2bba29d4c91bf3e8752eab93430 2023-03-24 01:54:13,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/dd817216eb45424b81a8b4d3b061c269 2023-03-24 01:54:13,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6a4d5160f7fa428382e9dfc3851887bb 2023-03-24 01:54:13,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5dc5cf4c55b54bd98b788de678bb1662 2023-03-24 01:54:13,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9e96bd741b3d46b4b25c62a86744d840 2023-03-24 01:54:13,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/492ae215a4cc487f992658d16824ee24 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/492ae215a4cc487f992658d16824ee24 2023-03-24 01:54:13,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c6bb879aab6044d985f3f443f08a5d20 2023-03-24 01:54:13,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/bb3d25f5275548738940515ab2504909 2023-03-24 01:54:13,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/cb45e44cc0f6493eae54b34055cedcfc 2023-03-24 01:54:13,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/4da25771a15e47d380363fd998446286 2023-03-24 01:54:13,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/25974c6b232d4e51bddf4093b644234e 2023-03-24 01:54:13,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/305d04c44a394799b5941c45fd301bce to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/305d04c44a394799b5941c45fd301bce 2023-03-24 01:54:13,681 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/f00bee47d43f4328b5ab9b5f1cabcca0 2023-03-24 01:54:13,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2032efa0aeef49bfaff52d123aff4510 2023-03-24 01:54:13,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/10baae9f7b4e4aef9c98583bc93ee302 2023-03-24 01:54:13,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/3f75ced0c49f4e5899cf201ac7d4afc4 2023-03-24 01:54:13,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a48bad59a094aeea4285bd391f25a58 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a48bad59a094aeea4285bd391f25a58 2023-03-24 01:54:13,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/c1e88b492f6e4a46a6de279ffb539f48 2023-03-24 01:54:13,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/454e389b0a6949698b0cf4420f2b3852 2023-03-24 01:54:13,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/062c157c29fe417f8bcd52b3f2bad130 2023-03-24 01:54:13,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/499ae78a53fa44c2b86b3ff67e6a9e0d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/499ae78a53fa44c2b86b3ff67e6a9e0d 2023-03-24 01:54:13,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/5970bcb324d749929ccac032bfa5de58 2023-03-24 01:54:13,693 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/0f4c083f63d24815a571fc6c3bfbdaad 2023-03-24 01:54:13,694 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a01c76c35d749cdaff88c2cac4bc8dc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2a01c76c35d749cdaff88c2cac4bc8dc 2023-03-24 01:54:13,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/2161bb1138cd4c2e96af37c034c2df91 2023-03-24 01:54:13,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/6469cf5b2dbf49328ef9aadaf6fb6602 2023-03-24 01:54:13,697 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/78125248c0de4515b42d1f4512321650 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/78125248c0de4515b42d1f4512321650 2023-03-24 01:54:13,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/8c3deb445006441eb5508f84630134ae 2023-03-24 01:54:13,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/98bcb6b2b82241ec8fb5cb059df80e9b 2023-03-24 01:54:13,699 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9330baec5dcd4a7e8c903aafb6d38898 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/9330baec5dcd4a7e8c903aafb6d38898 2023-03-24 01:54:13,701 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/d4f30c6ac18e44c9b1ca344d4945a05b 2023-03-24 01:54:13,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/33ee5dc740854809821acf8f7d55bfe1 2023-03-24 01:54:13,703 DEBUG [StoreCloser-TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/14153078dfac4797ab3c105089a1d682 2023-03-24 01:54:13,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/recovered.edits/1153.seqid, newMaxSeqId=1153, maxSeqId=1 2023-03-24 01:54:13,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf. 2023-03-24 01:54:13,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 63c1038f18949b69fb052a65a9d44cbf: 2023-03-24 01:54:13,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:13,711 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=63c1038f18949b69fb052a65a9d44cbf, regionState=CLOSED 2023-03-24 01:54:13,711 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622853711"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622853711"}]},"ts":"1679622853711"} 2023-03-24 01:54:13,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-24 01:54:13,714 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 63c1038f18949b69fb052a65a9d44cbf, server=jenkins-hbase4.apache.org,44439,1679622817494 in 1.5940 sec 2023-03-24 01:54:13,715 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-24 01:54:13,715 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=63c1038f18949b69fb052a65a9d44cbf, UNASSIGN in 1.5990 sec 2023-03-24 01:54:13,716 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622853716"}]},"ts":"1679622853716"} 2023-03-24 01:54:13,717 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:54:13,720 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:54:13,722 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6170 sec 2023-03-24 01:54:14,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 01:54:14,216 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-24 01:54:14,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:54:14,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,228 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,230 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 01:54:14,234 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:14,237 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/recovered.edits] 2023-03-24 01:54:14,239 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9aa6cae8c2b3452a8707051809b22fee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/9aa6cae8c2b3452a8707051809b22fee 2023-03-24 01:54:14,241 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fcbf7be61ecb445fb974e7467947a145 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/A/fcbf7be61ecb445fb974e7467947a145 2023-03-24 01:54:14,243 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/9df7018104094fdaba80b29b54d042f7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/9df7018104094fdaba80b29b54d042f7 2023-03-24 01:54:14,244 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d8f9dfd36ed54e6d9128542a190fd2d5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/B/d8f9dfd36ed54e6d9128542a190fd2d5 2023-03-24 01:54:14,246 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/23c16efbc5bb419cbad591f0eacb1913 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/23c16efbc5bb419cbad591f0eacb1913 2023-03-24 01:54:14,247 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/71956147e2fa4fc38e2bd9841a1c2ba5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/C/71956147e2fa4fc38e2bd9841a1c2ba5 2023-03-24 01:54:14,250 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/recovered.edits/1153.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf/recovered.edits/1153.seqid 2023-03-24 01:54:14,250 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/63c1038f18949b69fb052a65a9d44cbf 2023-03-24 01:54:14,250 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:54:14,253 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,273 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:54:14,315 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:54:14,316 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,316 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:54:14,316 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679622854316"}]},"ts":"9223372036854775807"} 2023-03-24 01:54:14,318 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:54:14,318 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 63c1038f18949b69fb052a65a9d44cbf, NAME => 'TestAcidGuarantees,,1679622822229.63c1038f18949b69fb052a65a9d44cbf.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:54:14,319 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:54:14,319 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679622854319"}]},"ts":"9223372036854775807"} 2023-03-24 01:54:14,321 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:54:14,323 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 101 msec 2023-03-24 01:54:14,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 01:54:14,332 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-24 01:54:14,348 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2b20a9e4-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1871118126_15 at /127.0.0.1:56296 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x2b20a9e4-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2b20a9e4-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2b20a9e4-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:44439-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=557 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 256) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5834 (was 6462) 2023-03-24 01:54:14,358 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=557, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=169, AvailableMemoryMB=5834 2023-03-24 01:54:14,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:54:14,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:54:14,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:14,363 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:54:14,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-24 01:54:14,364 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:54:14,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:14,365 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:14,366 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 empty. 2023-03-24 01:54:14,366 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:14,366 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:54:14,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:14,774 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:54:14,776 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 38628ffe26cfdb42701ddddf336a55f7, NAME => 'TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:54:14,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 38628ffe26cfdb42701ddddf336a55f7, disabling compactions & flushes 2023-03-24 01:54:15,182 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. after waiting 0 ms 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,182 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,182 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:15,184 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:54:15,185 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622855185"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622855185"}]},"ts":"1679622855185"} 2023-03-24 01:54:15,187 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:54:15,187 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:54:15,188 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622855188"}]},"ts":"1679622855188"} 2023-03-24 01:54:15,189 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:54:15,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, ASSIGN}] 2023-03-24 01:54:15,194 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, ASSIGN 2023-03-24 01:54:15,195 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:54:15,346 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:15,346 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622855346"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622855346"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622855346"}]},"ts":"1679622855346"} 2023-03-24 01:54:15,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:15,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:15,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 38628ffe26cfdb42701ddddf336a55f7, NAME => 'TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:54:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:54:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,505 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,507 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A 2023-03-24 01:54:15,507 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A 2023-03-24 01:54:15,507 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:15,507 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName A 2023-03-24 01:54:15,508 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:15,508 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,509 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B 2023-03-24 01:54:15,509 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B 2023-03-24 01:54:15,509 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:15,509 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName B 2023-03-24 01:54:15,510 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:15,510 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,511 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C 2023-03-24 01:54:15,511 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C 2023-03-24 01:54:15,512 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:15,512 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName C 2023-03-24 01:54:15,512 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:15,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:54:15,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:15,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:54:15,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 38628ffe26cfdb42701ddddf336a55f7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59811500, jitterRate=-0.10873919725418091}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:54:15,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:15,521 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., pid=18, masterSystemTime=1679622855499 2023-03-24 01:54:15,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:15,524 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:15,525 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622855524"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622855524"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622855524"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622855524"}]},"ts":"1679622855524"} 2023-03-24 01:54:15,528 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-24 01:54:15,528 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 in 178 msec 2023-03-24 01:54:15,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-24 01:54:15,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, ASSIGN in 336 msec 2023-03-24 01:54:15,530 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:54:15,530 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622855530"}]},"ts":"1679622855530"} 2023-03-24 01:54:15,531 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:54:15,534 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:54:15,535 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1730 sec 2023-03-24 01:54:16,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 01:54:16,469 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-24 01:54:16,470 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x31436cf6 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:16,475 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33c943f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:16,476 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:16,481 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:16,483 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:54:16,486 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56692, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:54:16,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:54:16,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:54:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:16,927 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:54:16,928 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:54:16,928 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:54:16,932 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 01:54:16,944 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, REOPEN/MOVE}] 2023-03-24 01:54:16,945 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, REOPEN/MOVE 2023-03-24 01:54:16,946 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:16,946 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622856946"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622856946"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622856946"}]},"ts":"1679622856946"} 2023-03-24 01:54:16,947 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:17,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 38628ffe26cfdb42701ddddf336a55f7, disabling compactions & flushes 2023-03-24 01:54:17,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. after waiting 0 ms 2023-03-24 01:54:17,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 01:54:17,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:17,104 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 38628ffe26cfdb42701ddddf336a55f7 to self. 2023-03-24 01:54:17,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,106 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=CLOSED 2023-03-24 01:54:17,107 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622857106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622857106"}]},"ts":"1679622857106"} 2023-03-24 01:54:17,109 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-24 01:54:17,109 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 in 161 msec 2023-03-24 01:54:17,110 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=true 2023-03-24 01:54:17,260 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,261 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622857260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622857260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622857260"}]},"ts":"1679622857260"} 2023-03-24 01:54:17,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:17,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 38628ffe26cfdb42701ddddf336a55f7, NAME => 'TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,421 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,422 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A 2023-03-24 01:54:17,422 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A 2023-03-24 01:54:17,422 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:17,424 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName A 2023-03-24 01:54:17,426 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:17,427 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,428 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B 2023-03-24 01:54:17,428 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B 2023-03-24 01:54:17,428 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:17,428 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName B 2023-03-24 01:54:17,429 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:17,429 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,429 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C 2023-03-24 01:54:17,429 DEBUG [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C 2023-03-24 01:54:17,430 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:17,430 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 38628ffe26cfdb42701ddddf336a55f7 columnFamilyName C 2023-03-24 01:54:17,431 INFO [StoreOpener-38628ffe26cfdb42701ddddf336a55f7-1] regionserver.HStore(310): Store=38628ffe26cfdb42701ddddf336a55f7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:17,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:54:17,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 38628ffe26cfdb42701ddddf336a55f7; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62671134, jitterRate=-0.06612733006477356}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:54:17,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:17,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., pid=23, masterSystemTime=1679622857414 2023-03-24 01:54:17,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,439 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,439 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622857438"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622857438"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622857438"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622857438"}]},"ts":"1679622857438"} 2023-03-24 01:54:17,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-24 01:54:17,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 in 178 msec 2023-03-24 01:54:17,443 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-24 01:54:17,443 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, REOPEN/MOVE in 497 msec 2023-03-24 01:54:17,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-24 01:54:17,445 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-03-24 01:54:17,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 941 msec 2023-03-24 01:54:17,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-24 01:54:17,457 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x29f78b2c to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,462 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51c62b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,463 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x7bbdba17 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,467 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,467 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,471 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,471 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,477 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,477 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,480 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,481 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,484 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,484 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,488 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,489 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,492 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,492 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:17,496 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:17,501 DEBUG [hconnection-0x725b09f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,501 DEBUG [hconnection-0x53ecd8e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,501 DEBUG [hconnection-0x2d238d7d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,503 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39016, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,513 DEBUG [hconnection-0x78a93f1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,516 DEBUG [hconnection-0x4ddb6fbc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,516 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,516 DEBUG [hconnection-0xfcf95be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,515 DEBUG [hconnection-0xf53a7db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,514 DEBUG [hconnection-0x440e6f8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,513 DEBUG [hconnection-0x63edd3be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:17,521 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39028, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,521 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39052, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,524 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39030, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,526 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39076, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,526 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39072, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,527 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,529 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39082, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:17,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:17,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:17,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:17,542 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:17,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:17,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:17,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:17,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:17,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,547 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,547 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:17,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:17,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:17,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:17,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:17,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:17,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:17,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:17,562 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,562 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:17,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64570d27[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:17,565 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:17,565 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:17,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:17,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622917562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622917565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622917565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622917567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622917568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622857555, max timestamp=1679622857566Num uniques 859; 2023-03-24 01:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:17,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622857555, max timestamp=1679622857566Num uniques 859; 2023-03-24 01:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:17,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622857555, max timestamp=1679622857566Num uniques 859; 2023-03-24 01:54:17,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:17,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:17,579 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:17,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:17,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:17,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:17,579 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:17,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,589 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,589 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,589 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,589 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:17,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:17,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,590 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:17,590 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:17,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622917683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622917684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622917686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622917686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622917688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,690 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:17,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:17,708 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:17,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:17,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:17,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:17,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:17,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:17,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:17,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:17,748 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:17,748 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:17,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:17,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:17,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:17,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,758 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc84337[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:17,762 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:17,762 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,763 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:17,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:17,764 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,764 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:17,767 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:17,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:17,773 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:17,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:17,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:17,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:17,773 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:17,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,778 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,782 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,782 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:17,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:17,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:17,782 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,783 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,783 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622917788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622917788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622917788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622917789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622917791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,883 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:17,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:17,896 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:17,896 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,896 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:17,896 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:17,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,898 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,898 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,899 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,900 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:17,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:17,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:17,909 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:17,909 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:17,909 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:17,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:17,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:17,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:17,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:17,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:17,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eacd054[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:17,924 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:17,924 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,931 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:17,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:17,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:17,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,933 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:17,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:17,936 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:17,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:17,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:17,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:17,936 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:17,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:17,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:17,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:17,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:17,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:17,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:17,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,944 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:17,944 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:17,944 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:17,944 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:17,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:17,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:17,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:17,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622917989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622917990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622917991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622917992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:17,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:17,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622917994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,013 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242d1a4554e35748449d7b10fd7f268d18 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242d1a4554e35748449d7b10fd7f268d18 2023-03-24 01:54:18,027 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/a3596b1f664648f2a057ed5135a2d40b 2023-03-24 01:54:18,044 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,060 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,060 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,060 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,060 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ae8f1a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,085 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,085 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,094 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,094 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,194 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,201 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,204 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,220 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,221 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,221 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,221 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,221 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14cc9d1d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,224 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,227 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,234 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,234 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,234 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,234 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,234 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,244 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,244 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622918295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622918298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622918299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622918299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622918299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,344 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,368 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,388 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,388 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,388 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,388 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,392 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab3a548[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,395 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,395 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,395 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,395 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,422 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,422 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,424 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,424 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,425 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,434 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,435 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,435 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/51cb5961002b4502b0b7dba69a5398e3 2023-03-24 01:54:18,535 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,538 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,542 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,542 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,542 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,542 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3295fc3a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,548 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,548 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,550 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,555 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,555 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,557 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,557 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,558 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,563 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,563 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,664 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,667 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,668 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,668 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,670 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,671 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,671 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,672 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,672 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,672 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,672 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,674 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,674 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eff732d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,676 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,677 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,677 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,681 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,681 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,694 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,694 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,794 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,798 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,800 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,800 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,800 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,801 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,801 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,801 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:18,802 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:18,803 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:18,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622918803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:18,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e2b339a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:18,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:18,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622918804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622918804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622918805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:18,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622918807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:18,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:18,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:18,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:18,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:18,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:18,815 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:18,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:18,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:18,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:18,815 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:18,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,818 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:18,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:18,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:18,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:18,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:18,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:18,827 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:18,827 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:18,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/59b785473b1f413fa0c62b2ea2cf2a69 2023-03-24 01:54:18,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/a3596b1f664648f2a057ed5135a2d40b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b 2023-03-24 01:54:18,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b, entries=200, sequenceid=15, filesize=31.9 K 2023-03-24 01:54:18,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/51cb5961002b4502b0b7dba69a5398e3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3 2023-03-24 01:54:18,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 01:54:18,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/59b785473b1f413fa0c62b2ea2cf2a69 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69 2023-03-24 01:54:18,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 01:54:18,904 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 38628ffe26cfdb42701ddddf336a55f7 in 1362ms, sequenceid=15, compaction requested=false 2023-03-24 01:54:18,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:18,927 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:18,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:18,934 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:18,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:18,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:18,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,937 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:18,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:18,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:18,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:18,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:18,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:18,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:18,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:18,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:18,940 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 01:54:18,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:18,940 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:18,941 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:18,941 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:18,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:18,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:19,246 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:54:19,400 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324717e384337df4caf90006d471952a358 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324717e384337df4caf90006d471952a358 2023-03-24 01:54:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:19,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:19,819 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/74b0c5f691c64895bf5454af658a51b4 2023-03-24 01:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622919820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622919822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622919824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622919826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622919827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622919927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622919928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622919928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622919932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:19,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:19,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622919932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622920134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622920134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622920134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622920135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622920136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,239 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/0971c13ba610479db64f0150a287dcb9 2023-03-24 01:54:20,251 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/cd8f091d7f514f13948cb962c2a309ff 2023-03-24 01:54:20,256 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/74b0c5f691c64895bf5454af658a51b4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4 2023-03-24 01:54:20,261 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4, entries=150, sequenceid=50, filesize=25.1 K 2023-03-24 01:54:20,262 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/0971c13ba610479db64f0150a287dcb9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9 2023-03-24 01:54:20,270 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 01:54:20,271 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/cd8f091d7f514f13948cb962c2a309ff as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff 2023-03-24 01:54:20,276 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 01:54:20,277 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for 38628ffe26cfdb42701ddddf336a55f7 in 1337ms, sequenceid=50, compaction requested=false 2023-03-24 01:54:20,277 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:20,277 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:20,283 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,283 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,286 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:20,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@591540f2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:20,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:20,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:20,289 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:20,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,290 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:20,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:20,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:20,291 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:20,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:20,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:20,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:20,291 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:20,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:20,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:20,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:20,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:20,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:20,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:20,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:20,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:20,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,304 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:20,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:20,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:20,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:20,304 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:20,304 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:20,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:20,404 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:20,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:20,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:20,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:20,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:20,408 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:20,408 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:20,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:20,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:20,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:20,411 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:20,411 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:20,411 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,411 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:20,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:20,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:20,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:20,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:20,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:20,413 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:20,413 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:20,414 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:20,414 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:20,429 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ab6dd158decb4478bd8b422c56ad8c27 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ab6dd158decb4478bd8b422c56ad8c27 2023-03-24 01:54:20,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:20,438 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:20,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622920451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622860439, max timestamp=1679622860454Num uniques 859; 2023-03-24 01:54:20,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622860439, max timestamp=1679622860454Num uniques 859; 2023-03-24 01:54:20,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:54:20,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622860439, max timestamp=1679622860454Num uniques 859; 2023-03-24 01:54:20,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:54:20,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:54:20,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622920472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622920472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622920473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622920474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622920556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622920579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622920579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622920580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622920580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622920761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622920783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622920783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622920784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:20,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622920785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:20,837 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/0db46d1a0b1440e18fa208af1cf965df 2023-03-24 01:54:21,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622921070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622921090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622921090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622921091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622921091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,249 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/c092c4ec76144fd7a619d1a641d3e6af 2023-03-24 01:54:21,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622921578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622921593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622921595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622921596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:21,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622921597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,664 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/a12f54fa0a464beabb5053d580857768 2023-03-24 01:54:21,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/0db46d1a0b1440e18fa208af1cf965df as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df 2023-03-24 01:54:21,674 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df, entries=150, sequenceid=61, filesize=25.1 K 2023-03-24 01:54:21,675 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/c092c4ec76144fd7a619d1a641d3e6af as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af 2023-03-24 01:54:21,680 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af, entries=150, sequenceid=61, filesize=11.6 K 2023-03-24 01:54:21,681 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/a12f54fa0a464beabb5053d580857768 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768 2023-03-24 01:54:21,691 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768, entries=150, sequenceid=61, filesize=11.6 K 2023-03-24 01:54:21,692 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 38628ffe26cfdb42701ddddf336a55f7 in 1279ms, sequenceid=61, compaction requested=true 2023-03-24 01:54:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:21,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:21,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fc03c6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:21,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:21,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:21,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:21,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:21,700 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:21,703 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:21,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:21,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:21,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:21,703 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:21,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:21,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:21,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:21,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:21,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:21,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:21,718 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:21,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:21,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:21,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,718 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:21,718 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:21,818 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:21,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:21,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:21,822 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:21,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:21,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:21,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:21,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:21,824 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:21,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:21,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:21,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:21,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:21,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:21,826 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:21,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:21,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:21,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:21,827 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-24 01:54:21,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:21,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:21,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:21,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:21,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:21,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:21,864 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f8120a9f8d8e4f19a4fc5d5cde03df5a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f8120a9f8d8e4f19a4fc5d5cde03df5a 2023-03-24 01:54:21,876 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/bc2f1aa5f33e4281b7b1a12955e854a2 2023-03-24 01:54:21,912 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/c64fe54ccb3c467592e865646d6326b1 2023-03-24 01:54:21,931 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/a4864c72035446c493d0d1e9582efa37 2023-03-24 01:54:21,943 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/bc2f1aa5f33e4281b7b1a12955e854a2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2 2023-03-24 01:54:21,948 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2, entries=150, sequenceid=95, filesize=25.1 K 2023-03-24 01:54:21,949 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/c64fe54ccb3c467592e865646d6326b1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1 2023-03-24 01:54:21,959 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1, entries=150, sequenceid=95, filesize=11.6 K 2023-03-24 01:54:21,960 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/a4864c72035446c493d0d1e9582efa37 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37 2023-03-24 01:54:21,976 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37, entries=150, sequenceid=95, filesize=11.6 K 2023-03-24 01:54:21,977 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 38628ffe26cfdb42701ddddf336a55f7 in 150ms, sequenceid=95, compaction requested=true 2023-03-24 01:54:21,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:21,978 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:21,981 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:21,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ef766f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:21,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:21,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:21,985 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:21,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:21,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:21,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:21,990 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:21,992 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:21,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:21,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:21,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:21,993 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:21,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:21,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:21,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:22,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,001 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,001 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,101 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,118 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,120 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,124 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,124 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,124 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:22,124 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,124 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167f242b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,131 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,132 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,132 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,134 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,138 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,138 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,140 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,141 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,142 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,146 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,146 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,146 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,146 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,246 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,254 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,256 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:22,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,265 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651ee4c6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,275 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,275 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,278 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,279 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:22,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,290 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,289 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,290 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,390 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,414 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,418 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,418 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:22,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,423 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41583990[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,429 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,431 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,436 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,437 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,437 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,437 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,437 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,439 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,439 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,439 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,447 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,447 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,447 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,547 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,568 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,571 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,574 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,574 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,574 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,574 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:22,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4e73da[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,580 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,580 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,602 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,607 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,608 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:22,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:22,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:22,694 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303247b18dd0018bb4466bbe3d0e4c35e797a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b18dd0018bb4466bbe3d0e4c35e797a 2023-03-24 01:54:22,702 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/10ebe98fc2dc415ab1eb5259e73d34b5 2023-03-24 01:54:22,708 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,712 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,712 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/1bb53672acf84d7787c85b4ef18a09d5 2023-03-24 01:54:22,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,720 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,720 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,720 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:22,720 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622922686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679622862658, max timestamp=1679622862705Num uniques 859; 2023-03-24 01:54:22,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,726 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9559, totalHeapSize=21976, min timestamp=1679622862658, max timestamp=1679622862705Num uniques 859; 2023-03-24 01:54:22,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ff40e4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39329, new segment data size=9559, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:22,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679622862658, max timestamp=1679622862705Num uniques 859; 2023-03-24 01:54:22,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622922689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:22,728 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,742 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,742 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,744 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,744 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,744 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,750 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,750 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:22,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:22,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622922754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622922754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622922755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622922826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622922827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,851 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:22,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:22,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622922859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622922859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:22,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622922860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:22,871 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:22,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:22,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:22,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,874 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:22,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:22,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:22,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:22,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:22,887 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,887 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:22,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:22,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7655a069[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:22,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:22,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,896 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:22,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:22,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:22,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:22,898 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:22,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:22,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:22,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:22,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:22,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:22,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:22,904 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,904 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,904 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:22,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:22,907 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:22,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:22,911 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:22,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:22,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:22,912 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:22,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:22,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:22,912 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:22,912 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:23,012 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:23,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:23,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:23,018 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:23,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:23,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:23,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,020 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:23,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:23,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622923029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:23,036 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:23,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:23,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622923034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:23,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c1938a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:23,041 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:23,041 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:23,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:23,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,046 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:23,046 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,046 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:23,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:23,049 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:23,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:23,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:23,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:23,049 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:23,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:23,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:23,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,060 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:23,060 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:23,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:23,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622923061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622923061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622923071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/5324ee6a031a412682154a84dd12c2a8 2023-03-24 01:54:23,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/10ebe98fc2dc415ab1eb5259e73d34b5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5 2023-03-24 01:54:23,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5, entries=350, sequenceid=109, filesize=52.3 K 2023-03-24 01:54:23,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/1bb53672acf84d7787c85b4ef18a09d5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5 2023-03-24 01:54:23,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5, entries=150, sequenceid=109, filesize=11.6 K 2023-03-24 01:54:23,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/5324ee6a031a412682154a84dd12c2a8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8 2023-03-24 01:54:23,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8, entries=150, sequenceid=109, filesize=11.6 K 2023-03-24 01:54:23,153 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 38628ffe26cfdb42701ddddf336a55f7 in 499ms, sequenceid=109, compaction requested=true 2023-03-24 01:54:23,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:23,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:23,154 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:23,154 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:23,156 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 163434 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:23,156 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:23,157 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:23,157 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,157 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:23,157 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=159.6 K 2023-03-24 01:54:23,157 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,157 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=58.0 K 2023-03-24 01:54:23,157 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting a3596b1f664648f2a057ed5135a2d40b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622857535 2023-03-24 01:54:23,158 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 51cb5961002b4502b0b7dba69a5398e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622857535 2023-03-24 01:54:23,158 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 74b0c5f691c64895bf5454af658a51b4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622857681 2023-03-24 01:54:23,158 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0971c13ba610479db64f0150a287dcb9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622857681 2023-03-24 01:54:23,158 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0db46d1a0b1440e18fa208af1cf965df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679622859822 2023-03-24 01:54:23,159 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c092c4ec76144fd7a619d1a641d3e6af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679622859822 2023-03-24 01:54:23,159 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting bc2f1aa5f33e4281b7b1a12955e854a2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679622860471 2023-03-24 01:54:23,159 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c64fe54ccb3c467592e865646d6326b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679622860471 2023-03-24 01:54:23,159 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 10ebe98fc2dc415ab1eb5259e73d34b5, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862590 2023-03-24 01:54:23,160 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1bb53672acf84d7787c85b4ef18a09d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862643 2023-03-24 01:54:23,160 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:23,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:23,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:23,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:23,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:23,169 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:23,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:23,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:23,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,172 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:23,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:23,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:23,177 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:23,178 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:23,178 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:23,180 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#121 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:23,186 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#120 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:23,215 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241790d2bd84914d47af21e3bfdabef7e1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241790d2bd84914d47af21e3bfdabef7e1 2023-03-24 01:54:23,219 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/5c7706ed842047f0a36c67bc89d33835 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5c7706ed842047f0a36c67bc89d33835 2023-03-24 01:54:23,241 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into 5c7706ed842047f0a36c67bc89d33835(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:23,241 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:23,241 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=11, startTime=1679622863154; duration=0sec 2023-03-24 01:54:23,241 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:23,241 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:23,243 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:23,245 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:23,246 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,246 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=58.0 K 2023-03-24 01:54:23,281 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 59b785473b1f413fa0c62b2ea2cf2a69, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622857535 2023-03-24 01:54:23,282 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cd8f091d7f514f13948cb962c2a309ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622857681 2023-03-24 01:54:23,283 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a12f54fa0a464beabb5053d580857768, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679622859822 2023-03-24 01:54:23,283 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a4864c72035446c493d0d1e9582efa37, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679622860471 2023-03-24 01:54:23,284 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/7afd6afbb8b74596b432f079724a7a0a 2023-03-24 01:54:23,285 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5324ee6a031a412682154a84dd12c2a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862643 2023-03-24 01:54:23,297 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/ea104d189ab34df4b96ca40ffcd9fa55 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/ea104d189ab34df4b96ca40ffcd9fa55 2023-03-24 01:54:23,307 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into ea104d189ab34df4b96ca40ffcd9fa55(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:23,307 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:23,307 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=11, startTime=1679622863153; duration=0sec 2023-03-24 01:54:23,307 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:23,310 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#124 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:23,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:23,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622923350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622923355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622923365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622923367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622923378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622923456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622923462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622923659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622923664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,728 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/78f74502e18d42918d33dc019c13571e 2023-03-24 01:54:23,744 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/9a61ce5c9557436583806f0bcfd5f1d2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/9a61ce5c9557436583806f0bcfd5f1d2 2023-03-24 01:54:23,746 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/57ff0f5a56db48999a10063d31ac355f 2023-03-24 01:54:23,753 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 9a61ce5c9557436583806f0bcfd5f1d2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:23,753 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/7afd6afbb8b74596b432f079724a7a0a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a 2023-03-24 01:54:23,753 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:23,753 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=11, startTime=1679622863154; duration=0sec 2023-03-24 01:54:23,753 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:23,758 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a, entries=150, sequenceid=145, filesize=25.3 K 2023-03-24 01:54:23,759 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/78f74502e18d42918d33dc019c13571e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e 2023-03-24 01:54:23,769 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e, entries=150, sequenceid=145, filesize=11.8 K 2023-03-24 01:54:23,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/57ff0f5a56db48999a10063d31ac355f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f 2023-03-24 01:54:23,784 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f, entries=150, sequenceid=145, filesize=11.8 K 2023-03-24 01:54:23,785 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 38628ffe26cfdb42701ddddf336a55f7 in 607ms, sequenceid=145, compaction requested=false 2023-03-24 01:54:23,785 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:23,785 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:23,787 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:23,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee9fe2b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:23,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:23,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:23,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:23,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:23,793 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:23,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:23,798 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:23,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:23,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:23,798 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:23,798 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:23,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:23,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,811 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:23,811 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:23,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:23,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:23,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622863887, max timestamp=1679622863899Num uniques 859; 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679622863887, max timestamp=1679622863899Num uniques 859; 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622863887, max timestamp=1679622863899Num uniques 859; 2023-03-24 01:54:23,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:23,911 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:23,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:23,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:23,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:23,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:23,915 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:23,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622923912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:23,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:23,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622923913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622923915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:23,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:23,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:23,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:23,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:23,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:23,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:23,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:23,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:23,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:23,923 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,923 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:23,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108f6772[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:23,925 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:23,925 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,926 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,926 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:23,926 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:23,926 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:23,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:23,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:23,934 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:23,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:23,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:23,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:23,935 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:23,936 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,936 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:23,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,936 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:23,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:23,941 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:23,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:23,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:23,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:23,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,942 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,942 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:23,942 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:23,942 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:23,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:23,942 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:23,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622923965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:23,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622923968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622924016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622924017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622924019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,042 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,045 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,047 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,047 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,049 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,049 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,049 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,049 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b82cbe2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,057 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,059 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,059 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,064 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,069 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,068 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,069 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,169 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,172 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,174 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,174 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,176 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,176 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,176 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,176 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,178 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31254cdb[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,183 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,183 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,186 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,186 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,194 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,194 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622924220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622924221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622924224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,287 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032408f41380244048ab818e5727f201d29f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032408f41380244048ab818e5727f201d29f 2023-03-24 01:54:24,294 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,297 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,301 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,301 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,303 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,303 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,303 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,303 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,303 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@707cf472[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,314 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,315 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,319 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,319 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,324 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,324 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,324 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,329 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,332 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,332 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,433 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,436 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,440 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,440 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,440 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,440 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,442 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,442 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b23b9f8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,447 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,447 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,450 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,450 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,452 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,455 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,455 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,455 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,455 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,456 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,456 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622924468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622924472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622924526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622924528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:24,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622924529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,556 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,560 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,560 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,562 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,564 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,565 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,566 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,566 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4623aca5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,571 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,571 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,571 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,571 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,573 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,573 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,580 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,583 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,583 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,683 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,686 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,686 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,686 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,686 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,690 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,690 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,690 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,690 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,693 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=159, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/6c39f4bb53074a378a36ba3dd0cf667f 2023-03-24 01:54:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d7b732e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,695 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,698 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,704 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,704 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,715 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,715 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:24,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,815 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,818 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,820 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:24,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a33c397[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:24,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:24,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,840 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,840 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,940 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:24,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:24,943 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:24,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:24,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:24,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,945 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:24,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:24,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:24,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:24,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:24,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1120673f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:24,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:24,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:24,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:24,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:24,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:24,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:24,964 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:24,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:24,964 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:24,966 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:24,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:24,967 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:24,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:24,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:24,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:24,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:24,972 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:24,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:24,972 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:24,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:24,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:24,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:24,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:25,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622925032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622925033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622925033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,073 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:25,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:25,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:25,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:25,076 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:25,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:25,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:25,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:25,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:25,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:25,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:25,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:25,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:25,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:25,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:25,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:25,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:25,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:25,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:25,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:25,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:25,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:25,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@455b21f7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:25,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:25,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:25,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:25,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:25,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:25,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:25,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:25,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:25,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:25,097 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:25,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:25,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:25,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:25,097 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:25,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:25,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:25,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,100 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:25,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:25,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:25,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/a91a7052edc64503b2a3cafb6253619f 2023-03-24 01:54:25,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:25,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:25,107 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:25,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,107 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:25,107 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:25,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/b651ddd41e3e4d94af56bcaba38595f8 2023-03-24 01:54:25,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/6c39f4bb53074a378a36ba3dd0cf667f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f 2023-03-24 01:54:25,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f, entries=200, sequenceid=159, filesize=32.1 K 2023-03-24 01:54:25,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/a91a7052edc64503b2a3cafb6253619f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f 2023-03-24 01:54:25,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f, entries=150, sequenceid=159, filesize=11.8 K 2023-03-24 01:54:25,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/b651ddd41e3e4d94af56bcaba38595f8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8 2023-03-24 01:54:25,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8, entries=150, sequenceid=159, filesize=11.8 K 2023-03-24 01:54:25,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 38628ffe26cfdb42701ddddf336a55f7 in 1277ms, sequenceid=159, compaction requested=true 2023-03-24 01:54:25,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:25,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:25,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:25,146 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:25,146 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:25,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:25,147 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:25,147 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84700 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:25,147 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:25,147 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:25,148 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,148 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,148 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5c7706ed842047f0a36c67bc89d33835, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=35.3 K 2023-03-24 01:54:25,148 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/ea104d189ab34df4b96ca40ffcd9fa55, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=82.7 K 2023-03-24 01:54:25,148 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5c7706ed842047f0a36c67bc89d33835, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862643 2023-03-24 01:54:25,148 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ea104d189ab34df4b96ca40ffcd9fa55, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862643 2023-03-24 01:54:25,149 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 78f74502e18d42918d33dc019c13571e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679622862750 2023-03-24 01:54:25,149 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7afd6afbb8b74596b432f079724a7a0a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679622862750 2023-03-24 01:54:25,149 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a91a7052edc64503b2a3cafb6253619f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:25,149 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6c39f4bb53074a378a36ba3dd0cf667f, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:25,156 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#129 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:25,161 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#130 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:25,207 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:25,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:25,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:25,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:25,210 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:25,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:25,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:25,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:25,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:25,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:25,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:25,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:25,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:25,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:25,222 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,222 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:25,222 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=131.94 KB heapSize=361.16 KB 2023-03-24 01:54:25,222 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:25,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:25,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:25,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:25,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:25,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:25,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:25,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622925490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622925490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,571 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/ca7a91aa547e4b4aabbbe6442d254a72 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/ca7a91aa547e4b4aabbbe6442d254a72 2023-03-24 01:54:25,576 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into ca7a91aa547e4b4aabbbe6442d254a72(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:25,577 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:25,577 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=13, startTime=1679622865146; duration=0sec 2023-03-24 01:54:25,577 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:25,577 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:25,578 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36130 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:25,578 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:25,578 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:25,578 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/9a61ce5c9557436583806f0bcfd5f1d2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=35.3 K 2023-03-24 01:54:25,579 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9a61ce5c9557436583806f0bcfd5f1d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1679622862643 2023-03-24 01:54:25,579 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 57ff0f5a56db48999a10063d31ac355f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1679622862750 2023-03-24 01:54:25,580 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b651ddd41e3e4d94af56bcaba38595f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:25,583 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/6fcf91104ae9413094e04bbbea95034a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6fcf91104ae9413094e04bbbea95034a 2023-03-24 01:54:25,588 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#132 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622925594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,596 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into 6fcf91104ae9413094e04bbbea95034a(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:25,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,596 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:25,597 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=13, startTime=1679622865146; duration=0sec 2023-03-24 01:54:25,597 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:25,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622925596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,649 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324042361ab86d9408da609ead36fca78f3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324042361ab86d9408da609ead36fca78f3 2023-03-24 01:54:25,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622925795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:25,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622925799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,017 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/32d8e1fe0468433183339123639bb29c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/32d8e1fe0468433183339123639bb29c 2023-03-24 01:54:26,021 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 32d8e1fe0468433183339123639bb29c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:26,021 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:26,021 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=13, startTime=1679622865146; duration=0sec 2023-03-24 01:54:26,021 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:26,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622926040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622926040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622926041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,055 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/303bb5c482a64fc4b15922976e3bdb02 2023-03-24 01:54:26,070 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/2de2b6c89f7a468c9d60f7c3f8e13237 2023-03-24 01:54:26,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622926098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622926102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,486 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/07126a8badd34668a5eb89fbb1233e31 2023-03-24 01:54:26,491 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/303bb5c482a64fc4b15922976e3bdb02 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02 2023-03-24 01:54:26,496 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02, entries=150, sequenceid=192, filesize=25.3 K 2023-03-24 01:54:26,499 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/2de2b6c89f7a468c9d60f7c3f8e13237 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237 2023-03-24 01:54:26,505 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237, entries=150, sequenceid=192, filesize=11.8 K 2023-03-24 01:54:26,506 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/07126a8badd34668a5eb89fbb1233e31 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31 2023-03-24 01:54:26,511 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31, entries=150, sequenceid=192, filesize=11.8 K 2023-03-24 01:54:26,512 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~361.11 KB/369776, currentSize=60.38 KB/61830 for 38628ffe26cfdb42701ddddf336a55f7 in 1290ms, sequenceid=192, compaction requested=false 2023-03-24 01:54:26,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:26,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:26,513 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:26,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,515 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,516 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43fed5c3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:26,517 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:26,517 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,519 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:26,519 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,519 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:26,521 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:26,521 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,521 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:26,527 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:26,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:26,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:26,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:26,527 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:26,529 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,529 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,529 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,537 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,539 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,539 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,539 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:26,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:26,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:26,539 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,540 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:26,540 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:26,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:26,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:26,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:26,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622926632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622926633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,640 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:26,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:26,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:26,644 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:26,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:26,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:26,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,646 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,646 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,646 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,646 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:26,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:26,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:26,648 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,648 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:26,648 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:26,648 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:26,650 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70aeea12[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:26,657 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:26,657 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:26,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:26,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,660 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:26,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:26,663 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:26,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:26,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:26,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:26,663 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:26,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,666 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,671 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:26,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,671 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:26,671 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622926736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622926736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,772 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:26,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:26,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:26,775 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:26,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:26,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:26,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,777 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,777 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,778 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,778 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:26,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:26,780 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:26,782 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,783 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622562a7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:26,784 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:26,784 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,785 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:26,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:26,785 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,785 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:26,788 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:26,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:26,790 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:26,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:26,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:26,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:26,790 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:26,792 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,793 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,793 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,799 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:26,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:26,799 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:26,799 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:26,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,800 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,900 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:26,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:26,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:26,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:26,903 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:26,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:26,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:26,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:26,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:26,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:26,911 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:26,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:26,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:26,912 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:26,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:26,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,918 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:26,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e6db38d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:26,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:26,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,921 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:26,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:26,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,923 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:26,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:26,926 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:26,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:26,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:26,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:26,926 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:26,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:26,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:26,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:26,930 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:26,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:26,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:26,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:26,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:26,934 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:26,934 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:26,935 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:26,935 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622926940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:26,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622926940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,018 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032471ecefcffe7445ae9aa89dd2e5ebb0a6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471ecefcffe7445ae9aa89dd2e5ebb0a6 2023-03-24 01:54:27,034 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:27,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:27,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:27,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:27,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:27,037 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:27,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:27,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:27,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,040 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,040 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:27,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:27,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:27,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:27,041 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:27,042 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:27,044 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,044 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:27,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:27,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b88314e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:27,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:27,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,052 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:27,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:27,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:27,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:27,056 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:27,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:27,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:27,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:27,056 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:27,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,060 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:27,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:27,062 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,062 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:27,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:27,063 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,163 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:27,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:27,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:27,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:27,166 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:27,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:27,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:27,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,171 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:27,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:27,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:27,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:27,174 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:27,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:27,175 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c71bc57[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:27,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:27,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,178 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:27,178 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:27,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:27,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:27,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:27,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:27,182 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:27,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:27,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:27,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:27,183 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:27,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:27,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:27,189 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,190 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:27,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:27,190 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:27,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622927242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622927244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,290 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:27,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:27,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:27,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:27,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:27,293 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:27,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:27,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:27,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,296 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:27,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:27,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:27,306 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,306 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:27,306 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:27,306 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:27,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:27,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:27,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588df24[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:27,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:27,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,316 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:27,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,318 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:27,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:27,320 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:27,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:27,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:27,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:27,320 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:27,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,324 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:27,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,328 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,328 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,328 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:27,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:27,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,328 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:27,328 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:27,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,424 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=208, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/0b288989db174f418353bc4a2aa2b388 2023-03-24 01:54:27,428 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:27,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:27,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:27,432 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:27,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:27,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:27,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:27,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:27,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:27,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:27,436 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:27,436 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:27,437 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,437 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:27,437 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:27,437 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:27,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:27,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/388f5be312844a83a6248feb3b48c128 2023-03-24 01:54:27,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:27,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:27,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16eae5c3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:27,442 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:27,442 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,443 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:27,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:27,443 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:27,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:27,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:27,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:27,448 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:27,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:27,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:27,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:27,448 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:27,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:27,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,452 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:27,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ea6e4898ef0d47a88881973c3bea6d7e 2023-03-24 01:54:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/0b288989db174f418353bc4a2aa2b388 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:27,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:27,458 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:27,458 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:27,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:27,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388, entries=150, sequenceid=208, filesize=25.3 K 2023-03-24 01:54:27,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/388f5be312844a83a6248feb3b48c128 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128 2023-03-24 01:54:27,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128, entries=150, sequenceid=208, filesize=11.8 K 2023-03-24 01:54:27,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ea6e4898ef0d47a88881973c3bea6d7e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e 2023-03-24 01:54:27,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e, entries=150, sequenceid=208, filesize=11.8 K 2023-03-24 01:54:27,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 38628ffe26cfdb42701ddddf336a55f7 in 877ms, sequenceid=208, compaction requested=true 2023-03-24 01:54:27,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:27,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:27,479 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:27,479 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:27,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:27,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:27,480 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77962 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:27,480 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:27,480 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:27,480 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,481 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:27,481 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6fcf91104ae9413094e04bbbea95034a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=76.1 K 2023-03-24 01:54:27,481 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,481 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/ca7a91aa547e4b4aabbbe6442d254a72, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=35.5 K 2023-03-24 01:54:27,481 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6fcf91104ae9413094e04bbbea95034a, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:27,481 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ca7a91aa547e4b4aabbbe6442d254a72, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:27,481 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 303bb5c482a64fc4b15922976e3bdb02, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679622863911 2023-03-24 01:54:27,482 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0b288989db174f418353bc4a2aa2b388, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:27,482 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2de2b6c89f7a468c9d60f7c3f8e13237, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679622863911 2023-03-24 01:54:27,483 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 388f5be312844a83a6248feb3b48c128, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:27,495 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#139 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:27,505 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#138 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:27,558 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:27,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:27,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:27,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:27,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:27,561 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:27,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:27,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:27,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:27,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:27,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:27,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:27,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:27,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:27,565 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,565 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:27,565 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:54:27,565 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:27,565 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:27,566 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:27,566 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:27,566 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:27,566 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:27,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:27,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:27,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622927758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622927761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622927862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:27,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622927865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:27,924 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/0377e1b9b4244e618a8f82dd6051559b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0377e1b9b4244e618a8f82dd6051559b 2023-03-24 01:54:27,929 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into 0377e1b9b4244e618a8f82dd6051559b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:27,929 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:27,929 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=13, startTime=1679622867479; duration=0sec 2023-03-24 01:54:27,929 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:27,929 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:27,932 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:27,932 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:27,932 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:27,932 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/32d8e1fe0468433183339123639bb29c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=35.5 K 2023-03-24 01:54:27,933 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 32d8e1fe0468433183339123639bb29c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1679622863341 2023-03-24 01:54:27,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 07126a8badd34668a5eb89fbb1233e31, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679622863911 2023-03-24 01:54:27,935 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ea6e4898ef0d47a88881973c3bea6d7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:27,941 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#141 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:27,948 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/43d61e321d8341edb597c6bcecf6a411 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/43d61e321d8341edb597c6bcecf6a411 2023-03-24 01:54:27,955 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into 43d61e321d8341edb597c6bcecf6a411(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:27,955 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:27,955 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=13, startTime=1679622867479; duration=0sec 2023-03-24 01:54:27,955 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:27,955 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/a04853af62994df7a04b65c3547f04ec as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a04853af62994df7a04b65c3547f04ec 2023-03-24 01:54:27,963 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into a04853af62994df7a04b65c3547f04ec(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:27,963 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:27,963 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=13, startTime=1679622867479; duration=0sec 2023-03-24 01:54:27,964 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:27,984 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324892fe7c05a2148b9a61f0b0048eec29c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324892fe7c05a2148b9a61f0b0048eec29c 2023-03-24 01:54:28,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622928044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,048 DEBUG [Thread-838] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:28,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622928051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,056 DEBUG [Thread-832] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:28,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622928052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,058 DEBUG [Thread-834] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:28,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622928064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622928067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622928368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622928369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,395 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/26849256e3754e81821a02bc1647a301 2023-03-24 01:54:28,446 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/8ec0472c6645433eb67f73c176bb6287 2023-03-24 01:54:28,479 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ae28e0b6a2d64df0b723fa730ff07991 2023-03-24 01:54:28,485 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/26849256e3754e81821a02bc1647a301 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301 2023-03-24 01:54:28,493 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301, entries=150, sequenceid=227, filesize=25.3 K 2023-03-24 01:54:28,494 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/8ec0472c6645433eb67f73c176bb6287 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287 2023-03-24 01:54:28,522 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287, entries=150, sequenceid=227, filesize=11.8 K 2023-03-24 01:54:28,523 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ae28e0b6a2d64df0b723fa730ff07991 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991 2023-03-24 01:54:28,535 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991, entries=150, sequenceid=227, filesize=11.8 K 2023-03-24 01:54:28,536 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 38628ffe26cfdb42701ddddf336a55f7 in 971ms, sequenceid=227, compaction requested=false 2023-03-24 01:54:28,536 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:28,536 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:28,538 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:28,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce2a411[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:28,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:28,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:28,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:28,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:28,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:28,551 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:28,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:28,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:28,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:28,551 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:28,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:28,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,556 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:28,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:28,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:28,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:28,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:28,559 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:28,559 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:28,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:28,659 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:28,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:28,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:28,663 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:28,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:28,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:28,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:28,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:28,666 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:28,666 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:28,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:28,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:28,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:28,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:28,668 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:28,668 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:28,668 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:28,686 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f84547b389e94da6ab0271dd707ac2c9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f84547b389e94da6ab0271dd707ac2c9 2023-03-24 01:54:28,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:28,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:28,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622928895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:28,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:28,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622928897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622928998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622929000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,091 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/e0cb78bf393c42d8babf76237ed29ae5 2023-03-24 01:54:29,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622929202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622929202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,501 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/7d3b687fefc94eec8fc96ea3fdf31333 2023-03-24 01:54:29,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39024 deadline: 1679622929504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39028 deadline: 1679622929505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,913 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/7302f5762c854508a4ea9925f755e8e9 2023-03-24 01:54:29,918 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/e0cb78bf393c42d8babf76237ed29ae5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5 2023-03-24 01:54:29,922 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5, entries=150, sequenceid=243, filesize=25.3 K 2023-03-24 01:54:29,923 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/7d3b687fefc94eec8fc96ea3fdf31333 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333 2023-03-24 01:54:29,927 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333, entries=150, sequenceid=243, filesize=11.8 K 2023-03-24 01:54:29,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/7302f5762c854508a4ea9925f755e8e9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9 2023-03-24 01:54:29,932 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9, entries=150, sequenceid=243, filesize=11.8 K 2023-03-24 01:54:29,933 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 38628ffe26cfdb42701ddddf336a55f7 in 1265ms, sequenceid=243, compaction requested=true 2023-03-24 01:54:29,933 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:29,933 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,934 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:29,934 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:29,936 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:29,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:29,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db030de[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:29,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:29,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:29,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:29,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:29,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:29,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:29,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:29,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:29,944 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:29,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:29,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:29,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:29,944 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:29,949 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,949 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,949 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:29,949 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:29,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:29,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:29,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:29,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:29,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:29,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:29,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:29,953 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:29,953 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:29,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:29,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:29,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:29,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:29,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:29,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:30,009 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622868893, max timestamp=1679622870010Num uniques 859; 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622868893, max timestamp=1679622870010Num uniques 859; 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622868893, max timestamp=1679622870010Num uniques 859; 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:30,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:30,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:30,033 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e599a7d419f4459083f0a1b30e5bbfdc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e599a7d419f4459083f0a1b30e5bbfdc 2023-03-24 01:54:30,053 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,056 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,058 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,060 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,060 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,060 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,060 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,063 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e02a5b3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,066 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,067 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,070 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,070 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622870059, max timestamp=1679622870072Num uniques 859; 2023-03-24 01:54:30,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:30,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622870059, max timestamp=1679622870072Num uniques 859; 2023-03-24 01:54:30,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622870059, max timestamp=1679622870072Num uniques 859; 2023-03-24 01:54:30,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:54:30,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,081 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,081 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622930107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622930108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,181 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,186 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,190 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,190 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,190 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,190 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f879f4d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,200 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,200 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,202 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,202 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,207 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,208 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622930209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622930211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,308 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,312 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,317 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,317 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,318 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564c7a80[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,321 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,321 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,321 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,323 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,323 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,323 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,326 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,326 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,327 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,329 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,337 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,337 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622930412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622930415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,437 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,441 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,441 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,442 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,442 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/626bcfd9a72445d3b231cd839affe430 2023-03-24 01:54:30,443 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,445 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,445 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,445 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,445 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,446 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,447 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a268657[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,457 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,462 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,463 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,473 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,473 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,573 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,576 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,580 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,582 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@102959d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,586 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,590 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,591 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,598 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,598 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,699 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,702 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621c1f07[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,711 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,711 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,711 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,712 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,715 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622930714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,718 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622930716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,724 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,724 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,824 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,828 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,831 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,831 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,831 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,831 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,833 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b1f1be[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,841 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,841 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,841 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,845 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,845 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,852 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,852 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,852 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/6f3e862158684fcbb64f34c76db6876c 2023-03-24 01:54:30,952 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:30,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:30,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:30,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:30,956 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:30,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:30,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:30,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:30,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:30,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:30,960 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,960 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:30,960 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:30,960 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:30,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:30,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:30,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699bd121[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:30,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:30,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,965 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:30,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:30,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:30,966 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:30,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:30,969 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:30,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:30,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:30,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:30,969 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:30,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:30,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:30,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,973 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:30,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:30,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:30,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:30,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:30,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:30,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:30,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:30,976 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:30,976 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,976 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:30,976 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:30,976 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:30,976 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:30,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:31,076 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:31,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:31,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:31,079 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:31,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:31,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:31,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:31,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:31,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:31,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:31,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:31,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:31,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:31,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:31,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:31,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:31,084 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:31,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:31,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c200ff2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:31,087 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:31,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,088 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:31,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:31,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:31,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:31,093 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:31,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:31,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:31,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:31,093 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:31,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,095 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:31,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:31,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:31,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:31,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:31,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:31,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:31,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:31,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:31,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,099 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:31,099 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:31,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:31,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:31,199 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:31,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:31,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:31,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:31,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:31,202 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:31,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:31,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:31,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,204 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:31,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:31,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:31,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:31,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:31,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:31,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:31,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:31,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:31,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:31,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:31,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e46c356[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:31,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:31,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,214 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:31,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:31,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:31,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:31,217 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:31,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:31,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:31,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:31,218 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:31,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:31,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622931220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:31,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622931223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:31,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:31,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:31,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:31,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:31,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:31,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:31,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:31,238 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:31,238 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:31,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:31,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/6b86e4fdfe744cffaff114ab55cf90d3 2023-03-24 01:54:31,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/626bcfd9a72445d3b231cd839affe430 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430 2023-03-24 01:54:31,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430, entries=150, sequenceid=264, filesize=25.4 K 2023-03-24 01:54:31,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/6f3e862158684fcbb64f34c76db6876c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c 2023-03-24 01:54:31,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c, entries=150, sequenceid=264, filesize=11.9 K 2023-03-24 01:54:31,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/6b86e4fdfe744cffaff114ab55cf90d3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3 2023-03-24 01:54:31,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3, entries=150, sequenceid=264, filesize=11.9 K 2023-03-24 01:54:31,296 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=154.31 KB/158010 for 38628ffe26cfdb42701ddddf336a55f7 in 1287ms, sequenceid=264, compaction requested=true 2023-03-24 01:54:31,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:31,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:31,296 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:54:31,296 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:54:31,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:31,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:31,298 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104110 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:54:31,298 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:54:31,298 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:31,298 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:31,298 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,298 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,298 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/43d61e321d8341edb597c6bcecf6a411, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=101.7 K 2023-03-24 01:54:31,298 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0377e1b9b4244e618a8f82dd6051559b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=47.5 K 2023-03-24 01:54:31,299 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 43d61e321d8341edb597c6bcecf6a411, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:31,299 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0377e1b9b4244e618a8f82dd6051559b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:31,299 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 26849256e3754e81821a02bc1647a301, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679622866621 2023-03-24 01:54:31,299 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8ec0472c6645433eb67f73c176bb6287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679622866621 2023-03-24 01:54:31,300 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e0cb78bf393c42d8babf76237ed29ae5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679622867756 2023-03-24 01:54:31,300 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7d3b687fefc94eec8fc96ea3fdf31333, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679622867756 2023-03-24 01:54:31,300 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 626bcfd9a72445d3b231cd839affe430, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:31,301 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6f3e862158684fcbb64f34c76db6876c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:31,313 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#151 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:31,316 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#150 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:31,338 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:31,338 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/cd6ba75f0d1749e19f3b8acb68235db4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/cd6ba75f0d1749e19f3b8acb68235db4 2023-03-24 01:54:31,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:31,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:31,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:31,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:31,346 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:31,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:31,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:31,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,348 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into cd6ba75f0d1749e19f3b8acb68235db4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:31,348 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:31,348 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=12, startTime=1679622871296; duration=0sec 2023-03-24 01:54:31,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,348 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:31,348 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:54:31,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:31,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:31,350 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48670 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:54:31,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:31,350 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:31,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:31,350 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:31,350 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a04853af62994df7a04b65c3547f04ec, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=47.5 K 2023-03-24 01:54:31,351 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a04853af62994df7a04b65c3547f04ec, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1679622865485 2023-03-24 01:54:31,351 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ae28e0b6a2d64df0b723fa730ff07991, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1679622866621 2023-03-24 01:54:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,352 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7302f5762c854508a4ea9925f755e8e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679622867756 2023-03-24 01:54:31,352 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6b86e4fdfe744cffaff114ab55cf90d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:31,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:31,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:31,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:31,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:31,356 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-24 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:31,356 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:31,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:31,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:31,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:31,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:31,375 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#152 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:31,757 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/00526f0585fb4ff5ad693a65a65db772 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00526f0585fb4ff5ad693a65a65db772 2023-03-24 01:54:31,766 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into 00526f0585fb4ff5ad693a65a65db772(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:31,766 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:31,766 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=12, startTime=1679622871296; duration=0sec 2023-03-24 01:54:31,766 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:31,797 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245ac5294591de465b882c596f24c8bd7d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245ac5294591de465b882c596f24c8bd7d 2023-03-24 01:54:31,799 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/05be51a713524919ab447e710736fddc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/05be51a713524919ab447e710736fddc 2023-03-24 01:54:31,804 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 05be51a713524919ab447e710736fddc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:31,804 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:31,804 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=12, startTime=1679622871296; duration=0sec 2023-03-24 01:54:31,804 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:32,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:32,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:32,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39030 deadline: 1679622932065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,071 DEBUG [Thread-834] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:32,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622932068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39016 deadline: 1679622932086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,089 DEBUG [Thread-832] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:32,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39052 deadline: 1679622932173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,218 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/7095d8d47d924189856eddb69aaa75ae 2023-03-24 01:54:32,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622932226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622932230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,260 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/26619475ee5046beb7ab29da4cfe5945 2023-03-24 01:54:32,295 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/3c00c795072f4a509e413ee7c97b85c1 2023-03-24 01:54:32,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/7095d8d47d924189856eddb69aaa75ae as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae 2023-03-24 01:54:32,305 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae, entries=150, sequenceid=302, filesize=25.4 K 2023-03-24 01:54:32,310 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/26619475ee5046beb7ab29da4cfe5945 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945 2023-03-24 01:54:32,318 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945, entries=150, sequenceid=302, filesize=11.9 K 2023-03-24 01:54:32,319 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/3c00c795072f4a509e413ee7c97b85c1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1 2023-03-24 01:54:32,326 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1, entries=150, sequenceid=302, filesize=11.9 K 2023-03-24 01:54:32,328 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for 38628ffe26cfdb42701ddddf336a55f7 in 972ms, sequenceid=302, compaction requested=false 2023-03-24 01:54:32,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:32,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:32,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:32,334 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:32,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a2e121[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:32,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:32,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:32,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:32,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:32,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:32,346 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:32,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:32,351 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:32,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:32,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:32,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:32,351 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:32,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:32,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:32,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,360 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:32,360 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:32,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:32,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:32,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:32,381 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:32,421 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324627f0ed605a14e61b287a88178ecc5e5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324627f0ed605a14e61b287a88178ecc5e5 2023-03-24 01:54:32,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622872433, max timestamp=1679622872456Num uniques 859; 2023-03-24 01:54:32,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:32,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622872433, max timestamp=1679622872456Num uniques 859; 2023-03-24 01:54:32,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:32,460 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:32,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622872433, max timestamp=1679622872456Num uniques 859; 2023-03-24 01:54:32,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:32,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:32,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:32,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:32,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:32,468 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:32,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:32,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:32,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:32,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:32,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:32,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:32,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:32,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:32,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:32,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:32,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:32,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:32,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:32,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11072e0d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:32,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:32,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:32,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:32,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:32,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:32,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:32,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:32,494 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:32,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:32,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:32,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:32,494 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:32,499 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,499 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,499 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:32,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:32,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,512 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:32,512 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:32,512 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,512 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,512 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,512 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,512 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622932520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,612 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:32,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:32,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:32,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:32,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:32,616 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:32,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:32,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:32,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,617 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,618 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,618 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:32,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:32,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:32,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:32,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:32,620 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,620 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:32,620 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:32,620 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:32,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:32,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:32,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:32,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d24fc40[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:32,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622932625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,629 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:32,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:32,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:32,631 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:32,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:32,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:32,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:32,631 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:32,633 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,633 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,633 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,638 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:32,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:32,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,639 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,639 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:32,739 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:32,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:32,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:32,742 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:32,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:32,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:32,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,744 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:32,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:32,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:32,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:32,745 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:32,746 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,746 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:32,746 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:32,746 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:32,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:32,748 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,748 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,748 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,748 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:32,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:32,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b0cec1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:32,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,751 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:32,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:32,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:32,755 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:32,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:32,755 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:32,757 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,757 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,757 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,758 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,759 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,761 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,761 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:32,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:32,761 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:32,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,761 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,761 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,762 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:32,762 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:32,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:32,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622932830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,857 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/a6385d327c3445c2a3fc282a3e86847d 2023-03-24 01:54:32,862 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:32,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:32,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:32,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:32,865 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:32,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:32,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:32,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,868 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,868 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,868 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/8d77a6441f1a460b90b8d718b6a24aa7 2023-03-24 01:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:32,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:32,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:32,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:32,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:32,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:32,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:32,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:32,870 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,870 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:32,871 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:32,871 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,871 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:32,871 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:32,871 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,871 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:32,871 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:32,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,873 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,873 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:32,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142a88bf[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:32,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:32,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:32,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:32,878 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,878 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:32,879 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:32,879 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,879 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:32,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:32,883 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:32,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:32,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:32,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:32,883 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:32,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/f9ef1b9a90234bdd805c88940c92be66 2023-03-24 01:54:32,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/a6385d327c3445c2a3fc282a3e86847d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d 2023-03-24 01:54:32,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:32,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d, entries=200, sequenceid=314, filesize=32.3 K 2023-03-24 01:54:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/8d77a6441f1a460b90b8d718b6a24aa7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7 2023-03-24 01:54:32,894 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,896 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7, entries=150, sequenceid=314, filesize=11.9 K 2023-03-24 01:54:32,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:32,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:32,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:32,899 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:32,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:32,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:32,899 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:32,899 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:32,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:32,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:32,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/f9ef1b9a90234bdd805c88940c92be66 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66 2023-03-24 01:54:32,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66, entries=150, sequenceid=314, filesize=11.9 K 2023-03-24 01:54:32,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 38628ffe26cfdb42701ddddf336a55f7 in 525ms, sequenceid=314, compaction requested=true 2023-03-24 01:54:32,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:32,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:32,906 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:32,906 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:32,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:32,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:32,907 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85690 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:32,907 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:32,907 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,908 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00526f0585fb4ff5ad693a65a65db772, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=83.7 K 2023-03-24 01:54:32,916 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:32,917 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:32,917 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,917 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 00526f0585fb4ff5ad693a65a65db772, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:32,917 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/cd6ba75f0d1749e19f3b8acb68235db4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=36.2 K 2023-03-24 01:54:32,917 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7095d8d47d924189856eddb69aaa75ae, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679622870102 2023-03-24 01:54:32,918 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cd6ba75f0d1749e19f3b8acb68235db4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:32,918 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting a6385d327c3445c2a3fc282a3e86847d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:32,919 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 26619475ee5046beb7ab29da4cfe5945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679622870102 2023-03-24 01:54:32,920 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8d77a6441f1a460b90b8d718b6a24aa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:32,932 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:32,937 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#159 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:32,957 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/b58be86de0b1455c9146acdcbdba75ed as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b58be86de0b1455c9146acdcbdba75ed 2023-03-24 01:54:32,966 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into b58be86de0b1455c9146acdcbdba75ed(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:32,966 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:32,966 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=13, startTime=1679622872906; duration=0sec 2023-03-24 01:54:32,966 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:32,966 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:32,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:32,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:32,968 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:32,968 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/05be51a713524919ab447e710736fddc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=36.2 K 2023-03-24 01:54:32,969 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/78bad769526b41dd849ce8a436c50327 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/78bad769526b41dd849ce8a436c50327 2023-03-24 01:54:32,969 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 05be51a713524919ab447e710736fddc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1679622868895 2023-03-24 01:54:32,970 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3c00c795072f4a509e413ee7c97b85c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679622870102 2023-03-24 01:54:32,970 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f9ef1b9a90234bdd805c88940c92be66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:32,983 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into 78bad769526b41dd849ce8a436c50327(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:32,983 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:32,983 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=13, startTime=1679622872906; duration=0sec 2023-03-24 01:54:32,983 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:32,992 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:32,999 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:33,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:33,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:33,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:33,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:33,003 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:33,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:33,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:33,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:33,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,004 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:33,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:33,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:33,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:33,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:33,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:33,008 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:33,008 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:33,009 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:33,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:33,033 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324603ed262a7e548b581c1b2327739959d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324603ed262a7e548b581c1b2327739959d 2023-03-24 01:54:33,050 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=353, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/e814b9aaa69d44a3bf679ff44f0fdc49 2023-03-24 01:54:33,109 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/0605a45d0dd24b658dc2bf2b0c3415b5 2023-03-24 01:54:33,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:33,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:33,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:33,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622933166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:33,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622933271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,402 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/59fd29387b8d423b8a3805abc503ea0f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59fd29387b8d423b8a3805abc503ea0f 2023-03-24 01:54:33,407 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 59fd29387b8d423b8a3805abc503ea0f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:33,407 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:33,407 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=13, startTime=1679622872906; duration=0sec 2023-03-24 01:54:33,407 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:33,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:33,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622933477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,554 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/708c182d0a524672a4de44f49d400314 2023-03-24 01:54:33,562 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/e814b9aaa69d44a3bf679ff44f0fdc49 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49 2023-03-24 01:54:33,566 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49, entries=150, sequenceid=353, filesize=25.4 K 2023-03-24 01:54:33,567 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/0605a45d0dd24b658dc2bf2b0c3415b5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5 2023-03-24 01:54:33,572 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5, entries=150, sequenceid=353, filesize=11.9 K 2023-03-24 01:54:33,573 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/708c182d0a524672a4de44f49d400314 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314 2023-03-24 01:54:33,576 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314, entries=150, sequenceid=353, filesize=11.9 K 2023-03-24 01:54:33,577 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 38628ffe26cfdb42701ddddf336a55f7 in 568ms, sequenceid=353, compaction requested=false 2023-03-24 01:54:33,577 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:33,577 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:33,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:33,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:33,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:33,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:33,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf5bb71[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:33,581 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:33,581 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:33,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:33,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:33,587 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:33,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:33,589 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:33,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:33,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:33,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:33,589 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:33,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:33,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:33,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:33,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:33,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:33,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:33,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:33,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:33,599 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:33,599 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:33,699 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:33,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:33,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:33,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:33,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:33,702 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:33,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:33,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:33,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:33,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:33,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:33,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:33,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:33,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:33,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:33,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:33,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:33,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:33,706 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:33,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:33,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:33,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:33,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:33,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:33,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:33,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:33,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:33,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622873803, max timestamp=1679622873819Num uniques 859; 2023-03-24 01:54:33,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:33,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622873803, max timestamp=1679622873819Num uniques 859; 2023-03-24 01:54:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622873803, max timestamp=1679622873819Num uniques 859; 2023-03-24 01:54:33,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:33,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:33,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622933872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:33,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:33,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622933975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,126 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248572a559dff6445c87123d5b7dab3bd4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248572a559dff6445c87123d5b7dab3bd4 2023-03-24 01:54:34,138 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/50ddbe627bcc49d2891d1509b5a89b9d 2023-03-24 01:54:34,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:34,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622934177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622934231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,235 DEBUG [Thread-836] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:34,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:34,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622934250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,252 DEBUG [Thread-840] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:34,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:34,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622934482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,577 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/b8d10dd7404c4d1bbabe682854e18b28 2023-03-24 01:54:34,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622934986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:34,993 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/6f0f87bf4266401c8522c3074bedea6e 2023-03-24 01:54:34,998 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/50ddbe627bcc49d2891d1509b5a89b9d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d 2023-03-24 01:54:35,002 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d, entries=150, sequenceid=364, filesize=25.4 K 2023-03-24 01:54:35,003 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/b8d10dd7404c4d1bbabe682854e18b28 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28 2023-03-24 01:54:35,007 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28, entries=150, sequenceid=364, filesize=11.9 K 2023-03-24 01:54:35,008 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/6f0f87bf4266401c8522c3074bedea6e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e 2023-03-24 01:54:35,012 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e, entries=150, sequenceid=364, filesize=11.9 K 2023-03-24 01:54:35,013 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 38628ffe26cfdb42701ddddf336a55f7 in 1307ms, sequenceid=364, compaction requested=true 2023-03-24 01:54:35,013 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:35,013 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:35,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,016 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:35,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:35,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f99653[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:35,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:35,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,019 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:35,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:35,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:35,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:35,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:35,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:35,024 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:35,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:35,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:35,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:35,024 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:35,025 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,025 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,026 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:35,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:35,026 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:35,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:35,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:35,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:35,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,030 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:35,030 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:35,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:35,030 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,030 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:35,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:35,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:35,030 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:35,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:35,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:35,030 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:35,030 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:35,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:35,130 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:35,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:35,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:35,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:35,134 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:35,136 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:35,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:35,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:35,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:35,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:35,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:35,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:35,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:35,138 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:35,138 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:35,156 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032480ffd40e8d88409e83167c219218b32c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032480ffd40e8d88409e83167c219218b32c 2023-03-24 01:54:35,161 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/01de9d376f0f48f397e80a8882a06af1 2023-03-24 01:54:35,587 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/746e468d81144f7eb0c51ed0883ee5c8 2023-03-24 01:54:36,004 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/2bba103436d649c594c93d8176523b7a 2023-03-24 01:54:36,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:36,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,009 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/01de9d376f0f48f397e80a8882a06af1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1 2023-03-24 01:54:36,013 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1, entries=150, sequenceid=400, filesize=25.4 K 2023-03-24 01:54:36,014 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/746e468d81144f7eb0c51ed0883ee5c8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8 2023-03-24 01:54:36,022 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8, entries=150, sequenceid=400, filesize=11.9 K 2023-03-24 01:54:36,023 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/2bba103436d649c594c93d8176523b7a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a 2023-03-24 01:54:36,028 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a, entries=150, sequenceid=400, filesize=11.9 K 2023-03-24 01:54:36,029 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=40.25 KB/41220 for 38628ffe26cfdb42701ddddf336a55f7 in 891ms, sequenceid=400, compaction requested=true 2023-03-24 01:54:36,029 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:36,029 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:36,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:36,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:36,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:36,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:36,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:36,033 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559c988b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,037 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,040 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,044 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,044 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,045 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324859dea6a8a304b559e965b0352b5d909 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324859dea6a8a304b559e965b0352b5d909 2023-03-24 01:54:36,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,047 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,061 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,061 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622876076, max timestamp=1679622876095Num uniques 859; 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622876076, max timestamp=1679622876095Num uniques 859; 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622876076, max timestamp=1679622876095Num uniques 859; 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:36,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:36,161 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:36,165 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622936161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,167 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,167 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,167 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,167 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,168 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,168 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,168 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,168 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,169 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,169 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,170 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b39df7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,172 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,173 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,174 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,176 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,181 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,187 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,187 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,187 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:36,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622936266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,287 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,290 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,294 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,294 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,294 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,294 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,295 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,295 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,295 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,295 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,295 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,296 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c9203[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,302 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,303 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,307 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,310 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,310 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,410 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,413 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,415 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,419 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e7dc7cf[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,421 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,421 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,423 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,424 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,429 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,429 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,431 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,436 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,436 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,436 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,436 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,464 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/bac4102d35cb4f8da45c9f02b1efc1d9 2023-03-24 01:54:36,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:36,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622936470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/1ef8f6ddddaa43858e150f7c7b6d2a47 2023-03-24 01:54:36,536 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,539 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,542 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,552 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f14584b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,555 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,559 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,559 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,570 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,570 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,570 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,570 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,570 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,570 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,670 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,673 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,679 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,679 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,680 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,680 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,685 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,685 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,685 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,685 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641207b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,692 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,694 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,694 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,695 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,695 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,695 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,700 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,700 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,700 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,700 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622936774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,800 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,803 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,805 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,807 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,807 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,807 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:36,807 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:36,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:36,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bdb63a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:36,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:36,815 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:36,815 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,815 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,815 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:36,817 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:36,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:36,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:36,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:36,817 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:36,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:36,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:36,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:36,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:36,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:36,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:36,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:36,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,823 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,823 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:36,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:36,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:36,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ddc1af4dbb7144dd96f1df57bb429944 2023-03-24 01:54:36,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/bac4102d35cb4f8da45c9f02b1efc1d9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9 2023-03-24 01:54:36,903 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9, entries=350, sequenceid=410, filesize=52.9 K 2023-03-24 01:54:36,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/1ef8f6ddddaa43858e150f7c7b6d2a47 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47 2023-03-24 01:54:36,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47, entries=150, sequenceid=410, filesize=11.9 K 2023-03-24 01:54:36,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ddc1af4dbb7144dd96f1df57bb429944 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944 2023-03-24 01:54:36,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944, entries=150, sequenceid=410, filesize=11.9 K 2023-03-24 01:54:36,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 38628ffe26cfdb42701ddddf336a55f7 in 894ms, sequenceid=410, compaction requested=true 2023-03-24 01:54:36,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:36,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:36,923 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:36,923 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:36,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:36,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:36,924 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:36,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:36,925 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:36,925 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159012 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:36,926 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:36,926 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:36,926 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,926 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,926 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b58be86de0b1455c9146acdcbdba75ed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=60.1 K 2023-03-24 01:54:36,926 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/78bad769526b41dd849ce8a436c50327, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=155.3 K 2023-03-24 01:54:36,927 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b58be86de0b1455c9146acdcbdba75ed, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:36,927 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 78bad769526b41dd849ce8a436c50327, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:36,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:36,927 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0605a45d0dd24b658dc2bf2b0c3415b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679622872497 2023-03-24 01:54:36,928 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e814b9aaa69d44a3bf679ff44f0fdc49, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679622872497 2023-03-24 01:54:36,928 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b8d10dd7404c4d1bbabe682854e18b28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679622873141 2023-03-24 01:54:36,928 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 50ddbe627bcc49d2891d1509b5a89b9d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679622873141 2023-03-24 01:54:36,929 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 746e468d81144f7eb0c51ed0883ee5c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622873856 2023-03-24 01:54:36,929 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 01de9d376f0f48f397e80a8882a06af1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622873856 2023-03-24 01:54:36,929 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting bac4102d35cb4f8da45c9f02b1efc1d9, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876005 2023-03-24 01:54:36,930 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1ef8f6ddddaa43858e150f7c7b6d2a47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876009 2023-03-24 01:54:36,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:36,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:36,936 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:36,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:36,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:36,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:36,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:36,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:36,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:36,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:36,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:36,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. started... 2023-03-24 01:54:36,942 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 01:54:36,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:36,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:36,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:36,943 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:36,943 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:36,943 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:36,945 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#174 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:36,970 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#175 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:36,979 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/9fafbc273136405dbd547acad025cca4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/9fafbc273136405dbd547acad025cca4 2023-03-24 01:54:36,986 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into 9fafbc273136405dbd547acad025cca4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:36,986 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:36,986 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=11, startTime=1679622876923; duration=0sec 2023-03-24 01:54:36,986 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:36,986 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:54:36,988 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:54:36,988 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:36,988 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:36,988 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59fd29387b8d423b8a3805abc503ea0f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=60.1 K 2023-03-24 01:54:36,988 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 59fd29387b8d423b8a3805abc503ea0f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1679622872063 2023-03-24 01:54:36,989 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 708c182d0a524672a4de44f49d400314, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679622872497 2023-03-24 01:54:36,990 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6f0f87bf4266401c8522c3074bedea6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679622873141 2023-03-24 01:54:36,991 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2bba103436d649c594c93d8176523b7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622873856 2023-03-24 01:54:36,991 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ddc1af4dbb7144dd96f1df57bb429944, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876009 2023-03-24 01:54:37,013 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#177 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:37,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:37,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. as already flushing 2023-03-24 01:54:37,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:37,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622937296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:37,384 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324cde1ea98fdf44b2cb90117686679bad1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cde1ea98fdf44b2cb90117686679bad1 2023-03-24 01:54:37,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622937399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:37,424 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/57fd90d6a9514626ad3cf762e330952b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57fd90d6a9514626ad3cf762e330952b 2023-03-24 01:54:37,429 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 57fd90d6a9514626ad3cf762e330952b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:37,429 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:37,429 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=11, startTime=1679622876923; duration=0sec 2023-03-24 01:54:37,429 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:37,431 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/00e2aa7ff9bf44d68fec7ae47f2caf9f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00e2aa7ff9bf44d68fec7ae47f2caf9f 2023-03-24 01:54:37,444 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into 00e2aa7ff9bf44d68fec7ae47f2caf9f(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:37,444 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:37,444 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=11, startTime=1679622876923; duration=0sec 2023-03-24 01:54:37,445 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:37,499 DEBUG [Thread-843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:62686 2023-03-24 01:54:37,499 DEBUG [Thread-843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:37,501 DEBUG [Thread-847] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:62686 2023-03-24 01:54:37,501 DEBUG [Thread-847] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:37,502 DEBUG [Thread-845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:62686 2023-03-24 01:54:37,502 DEBUG [Thread-845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:37,503 DEBUG [Thread-849] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:62686 2023-03-24 01:54:37,503 DEBUG [Thread-849] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:37,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:37,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622937604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:37,790 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=447, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/8fb09e6191964905869efe94c1521c40 2023-03-24 01:54:37,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:37,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622937905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,200 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/48b05bf2382743758534d19707c35bfd 2023-03-24 01:54:38,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:38,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39024 deadline: 1679622938244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,246 DEBUG [Thread-836] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:38,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:38,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39028 deadline: 1679622938278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,280 DEBUG [Thread-840] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:54:38,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:38,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39052 deadline: 1679622938406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,610 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/61de23f7301b4dde9a7ba9b1065574bb 2023-03-24 01:54:38,614 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/8fb09e6191964905869efe94c1521c40 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40 2023-03-24 01:54:38,617 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40, entries=150, sequenceid=447, filesize=25.4 K 2023-03-24 01:54:38,617 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/48b05bf2382743758534d19707c35bfd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd 2023-03-24 01:54:38,621 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd, entries=150, sequenceid=447, filesize=11.9 K 2023-03-24 01:54:38,622 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/61de23f7301b4dde9a7ba9b1065574bb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb 2023-03-24 01:54:38,625 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb, entries=150, sequenceid=447, filesize=11.9 K 2023-03-24 01:54:38,626 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 38628ffe26cfdb42701ddddf336a55f7 in 1684ms, sequenceid=447, compaction requested=false 2023-03-24 01:54:38,626 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:38,626 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:38,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:38,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:38,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:38,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:38,628 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,628 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:38,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:38,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@362e569d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:38,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:38,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:38,632 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,632 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:38,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:38,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:38,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:38,633 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:38,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:38,635 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:38,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:38,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:38,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:38,635 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:38,636 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,636 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:38,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:38,636 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:38,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:38,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:38,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:38,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:38,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:38,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:38,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:38,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:38,643 DEBUG [Thread-842] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:38,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:38,643 DEBUG [Thread-842] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:38,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:38,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:38,743 DEBUG [Thread-842] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:38,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:39,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:39,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:54:39,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:39,418 DEBUG [Thread-838] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:62686 2023-03-24 01:54:39,418 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:39,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:39,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:39,418 DEBUG [Thread-838] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:39,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:39,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:39,610 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 01:54:39,835 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240bc5f0ac48524361a5c9ed0eacb3bacf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc5f0ac48524361a5c9ed0eacb3bacf 2023-03-24 01:54:40,240 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/836d9b18736847b287804d5dcb06f8ee 2023-03-24 01:54:40,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/61f39a72290d4fb88cdfebf1abf85ec2 2023-03-24 01:54:40,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/b9f43086495e4b75b797dc895cd024f4 2023-03-24 01:54:40,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/836d9b18736847b287804d5dcb06f8ee as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee 2023-03-24 01:54:40,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee, entries=150, sequenceid=461, filesize=25.4 K 2023-03-24 01:54:40,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/61f39a72290d4fb88cdfebf1abf85ec2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2 2023-03-24 01:54:40,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2, entries=150, sequenceid=461, filesize=11.9 K 2023-03-24 01:54:40,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/b9f43086495e4b75b797dc895cd024f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4 2023-03-24 01:54:40,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4, entries=150, sequenceid=461, filesize=11.9 K 2023-03-24 01:54:40,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 38628ffe26cfdb42701ddddf336a55f7 in 1263ms, sequenceid=461, compaction requested=true 2023-03-24 01:54:40,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:40,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:40,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:40,681 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:40,681 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:40,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/A is initiating minor compaction (all files) 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:40,682 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/A in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/B is initiating minor compaction (all files) 2023-03-24 01:54:40,682 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00e2aa7ff9bf44d68fec7ae47f2caf9f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=77.1 K 2023-03-24 01:54:40,682 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/B in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:40,682 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/9fafbc273136405dbd547acad025cca4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=36.5 K 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 00e2aa7ff9bf44d68fec7ae47f2caf9f, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876009 2023-03-24 01:54:40,682 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9fafbc273136405dbd547acad025cca4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876009 2023-03-24 01:54:40,683 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8fb09e6191964905869efe94c1521c40, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622876147 2023-03-24 01:54:40,683 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 48b05bf2382743758534d19707c35bfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622876147 2023-03-24 01:54:40,683 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 836d9b18736847b287804d5dcb06f8ee, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679622877277 2023-03-24 01:54:40,683 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 61f39a72290d4fb88cdfebf1abf85ec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679622877277 2023-03-24 01:54:40,689 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#B#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:40,694 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#A#compaction#184 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:41,099 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/5b52e574878b42c3b7943fe7b97343a8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5b52e574878b42c3b7943fe7b97343a8 2023-03-24 01:54:41,104 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/B of 38628ffe26cfdb42701ddddf336a55f7 into 5b52e574878b42c3b7943fe7b97343a8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:41,104 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:41,104 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/B, priority=13, startTime=1679622880681; duration=0sec 2023-03-24 01:54:41,104 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:41,104 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:41,105 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:41,105 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 38628ffe26cfdb42701ddddf336a55f7/C is initiating minor compaction (all files) 2023-03-24 01:54:41,105 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 38628ffe26cfdb42701ddddf336a55f7/C in TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:41,105 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57fd90d6a9514626ad3cf762e330952b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp, totalSize=36.5 K 2023-03-24 01:54:41,106 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 57fd90d6a9514626ad3cf762e330952b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679622876009 2023-03-24 01:54:41,106 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 61de23f7301b4dde9a7ba9b1065574bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622876147 2023-03-24 01:54:41,106 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b9f43086495e4b75b797dc895cd024f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679622877277 2023-03-24 01:54:41,112 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 38628ffe26cfdb42701ddddf336a55f7#C#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:41,519 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/1430ebf26fe64cb99e9c82e8072a88b0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/1430ebf26fe64cb99e9c82e8072a88b0 2023-03-24 01:54:41,523 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/C of 38628ffe26cfdb42701ddddf336a55f7 into 1430ebf26fe64cb99e9c82e8072a88b0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:41,524 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:41,524 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/C, priority=13, startTime=1679622880681; duration=0sec 2023-03-24 01:54:41,524 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:41,913 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/b5166c553ba04d2a93dcc6ad166fcf12 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/b5166c553ba04d2a93dcc6ad166fcf12 2023-03-24 01:54:41,916 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 38628ffe26cfdb42701ddddf336a55f7/A of 38628ffe26cfdb42701ddddf336a55f7 into b5166c553ba04d2a93dcc6ad166fcf12(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 01:54:41,916 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:41,916 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7., storeName=38628ffe26cfdb42701ddddf336a55f7/A, priority=13, startTime=1679622880681; duration=1sec 2023-03-24 01:54:41,917 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:42,114 DEBUG [Thread-834] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bbdba17 to 127.0.0.1:62686 2023-03-24 01:54:42,115 DEBUG [Thread-834] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:42,170 DEBUG [Thread-832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29f78b2c to 127.0.0.1:62686 2023-03-24 01:54:42,171 DEBUG [Thread-832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:48,287 DEBUG [Thread-836] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:62686 2023-03-24 01:54:48,287 DEBUG [Thread-836] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:48,372 DEBUG [Thread-840] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:62686 2023-03-24 01:54:48,372 DEBUG [Thread-840] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 40 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4673 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4552 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:54:48,372 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1956 2023-03-24 01:54:48,373 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5868 rows 2023-03-24 01:54:48,373 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1954 2023-03-24 01:54:48,373 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5862 rows 2023-03-24 01:54:48,373 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:54:48,373 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31436cf6 to 127.0.0.1:62686 2023-03-24 01:54:48,373 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:54:48,374 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:54:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:54:48,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:48,379 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622888379"}]},"ts":"1679622888379"} 2023-03-24 01:54:48,380 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:54:48,385 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:54:48,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, UNASSIGN}] 2023-03-24 01:54:48,387 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, UNASSIGN 2023-03-24 01:54:48,388 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:48,388 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622888388"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622888388"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622888388"}]},"ts":"1679622888388"} 2023-03-24 01:54:48,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:48,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 38628ffe26cfdb42701ddddf336a55f7, disabling compactions & flushes 2023-03-24 01:54:48,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. after waiting 0 ms 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:48,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 38628ffe26cfdb42701ddddf336a55f7 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=A 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=B 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 38628ffe26cfdb42701ddddf336a55f7, store=C 2023-03-24 01:54:48,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:48,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:48,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240fcad6580bb04345980ebdc3dd0c4437 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240fcad6580bb04345980ebdc3dd0c4437 2023-03-24 01:54:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:49,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=471, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/6399b243cca54e48806797ca9596f0d9 2023-03-24 01:54:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:49,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/57256a6d093f42058f0d428d45a9e083 2023-03-24 01:54:50,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ce697f8271cf4888b8243c82e2286020 2023-03-24 01:54:50,184 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/A/6399b243cca54e48806797ca9596f0d9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6399b243cca54e48806797ca9596f0d9 2023-03-24 01:54:50,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6399b243cca54e48806797ca9596f0d9, entries=100, sequenceid=471, filesize=18.6 K 2023-03-24 01:54:50,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/B/57256a6d093f42058f0d428d45a9e083 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/57256a6d093f42058f0d428d45a9e083 2023-03-24 01:54:50,190 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/57256a6d093f42058f0d428d45a9e083, entries=100, sequenceid=471, filesize=9.5 K 2023-03-24 01:54:50,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/.tmp/C/ce697f8271cf4888b8243c82e2286020 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ce697f8271cf4888b8243c82e2286020 2023-03-24 01:54:50,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ce697f8271cf4888b8243c82e2286020, entries=100, sequenceid=471, filesize=9.5 K 2023-03-24 01:54:50,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 38628ffe26cfdb42701ddddf336a55f7 in 1654ms, sequenceid=471, compaction requested=false 2023-03-24 01:54:50,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/ea104d189ab34df4b96ca40ffcd9fa55, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6fcf91104ae9413094e04bbbea95034a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/43d61e321d8341edb597c6bcecf6a411, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00526f0585fb4ff5ad693a65a65db772, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/78bad769526b41dd849ce8a436c50327, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00e2aa7ff9bf44d68fec7ae47f2caf9f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee] to archive 2023-03-24 01:54:50,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:50,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a3596b1f664648f2a057ed5135a2d40b 2023-03-24 01:54:50,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/74b0c5f691c64895bf5454af658a51b4 2023-03-24 01:54:50,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0db46d1a0b1440e18fa208af1cf965df 2023-03-24 01:54:50,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bc2f1aa5f33e4281b7b1a12955e854a2 2023-03-24 01:54:50,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/10ebe98fc2dc415ab1eb5259e73d34b5 2023-03-24 01:54:50,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/ea104d189ab34df4b96ca40ffcd9fa55 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/ea104d189ab34df4b96ca40ffcd9fa55 2023-03-24 01:54:50,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7afd6afbb8b74596b432f079724a7a0a 2023-03-24 01:54:50,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6c39f4bb53074a378a36ba3dd0cf667f 2023-03-24 01:54:50,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6fcf91104ae9413094e04bbbea95034a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6fcf91104ae9413094e04bbbea95034a 2023-03-24 01:54:50,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/303bb5c482a64fc4b15922976e3bdb02 2023-03-24 01:54:50,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/43d61e321d8341edb597c6bcecf6a411 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/43d61e321d8341edb597c6bcecf6a411 2023-03-24 01:54:50,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/0b288989db174f418353bc4a2aa2b388 2023-03-24 01:54:50,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/26849256e3754e81821a02bc1647a301 2023-03-24 01:54:50,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e0cb78bf393c42d8babf76237ed29ae5 2023-03-24 01:54:50,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00526f0585fb4ff5ad693a65a65db772 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00526f0585fb4ff5ad693a65a65db772 2023-03-24 01:54:50,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/626bcfd9a72445d3b231cd839affe430 2023-03-24 01:54:50,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/7095d8d47d924189856eddb69aaa75ae 2023-03-24 01:54:50,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/a6385d327c3445c2a3fc282a3e86847d 2023-03-24 01:54:50,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/78bad769526b41dd849ce8a436c50327 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/78bad769526b41dd849ce8a436c50327 2023-03-24 01:54:50,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/e814b9aaa69d44a3bf679ff44f0fdc49 2023-03-24 01:54:50,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/50ddbe627bcc49d2891d1509b5a89b9d 2023-03-24 01:54:50,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/01de9d376f0f48f397e80a8882a06af1 2023-03-24 01:54:50,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/bac4102d35cb4f8da45c9f02b1efc1d9 2023-03-24 01:54:50,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00e2aa7ff9bf44d68fec7ae47f2caf9f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/00e2aa7ff9bf44d68fec7ae47f2caf9f 2023-03-24 01:54:50,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/8fb09e6191964905869efe94c1521c40 2023-03-24 01:54:50,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/836d9b18736847b287804d5dcb06f8ee 2023-03-24 01:54:50,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5c7706ed842047f0a36c67bc89d33835, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/ca7a91aa547e4b4aabbbe6442d254a72, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0377e1b9b4244e618a8f82dd6051559b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/cd6ba75f0d1749e19f3b8acb68235db4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b58be86de0b1455c9146acdcbdba75ed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/9fafbc273136405dbd547acad025cca4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2] to archive 2023-03-24 01:54:50,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:50,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/51cb5961002b4502b0b7dba69a5398e3 2023-03-24 01:54:50,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0971c13ba610479db64f0150a287dcb9 2023-03-24 01:54:50,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c092c4ec76144fd7a619d1a641d3e6af 2023-03-24 01:54:50,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/c64fe54ccb3c467592e865646d6326b1 2023-03-24 01:54:50,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5c7706ed842047f0a36c67bc89d33835 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5c7706ed842047f0a36c67bc89d33835 2023-03-24 01:54:50,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1bb53672acf84d7787c85b4ef18a09d5 2023-03-24 01:54:50,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/78f74502e18d42918d33dc019c13571e 2023-03-24 01:54:50,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/ca7a91aa547e4b4aabbbe6442d254a72 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/ca7a91aa547e4b4aabbbe6442d254a72 2023-03-24 01:54:50,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/a91a7052edc64503b2a3cafb6253619f 2023-03-24 01:54:50,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/2de2b6c89f7a468c9d60f7c3f8e13237 2023-03-24 01:54:50,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0377e1b9b4244e618a8f82dd6051559b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0377e1b9b4244e618a8f82dd6051559b 2023-03-24 01:54:50,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/388f5be312844a83a6248feb3b48c128 2023-03-24 01:54:50,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8ec0472c6645433eb67f73c176bb6287 2023-03-24 01:54:50,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/7d3b687fefc94eec8fc96ea3fdf31333 2023-03-24 01:54:50,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/cd6ba75f0d1749e19f3b8acb68235db4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/cd6ba75f0d1749e19f3b8acb68235db4 2023-03-24 01:54:50,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/6f3e862158684fcbb64f34c76db6876c 2023-03-24 01:54:50,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/26619475ee5046beb7ab29da4cfe5945 2023-03-24 01:54:50,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b58be86de0b1455c9146acdcbdba75ed to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b58be86de0b1455c9146acdcbdba75ed 2023-03-24 01:54:50,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/8d77a6441f1a460b90b8d718b6a24aa7 2023-03-24 01:54:50,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/0605a45d0dd24b658dc2bf2b0c3415b5 2023-03-24 01:54:50,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/b8d10dd7404c4d1bbabe682854e18b28 2023-03-24 01:54:50,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/746e468d81144f7eb0c51ed0883ee5c8 2023-03-24 01:54:50,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/9fafbc273136405dbd547acad025cca4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/9fafbc273136405dbd547acad025cca4 2023-03-24 01:54:50,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/1ef8f6ddddaa43858e150f7c7b6d2a47 2023-03-24 01:54:50,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/48b05bf2382743758534d19707c35bfd 2023-03-24 01:54:50,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/61f39a72290d4fb88cdfebf1abf85ec2 2023-03-24 01:54:50,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/9a61ce5c9557436583806f0bcfd5f1d2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/32d8e1fe0468433183339123639bb29c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a04853af62994df7a04b65c3547f04ec, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/05be51a713524919ab447e710736fddc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59fd29387b8d423b8a3805abc503ea0f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57fd90d6a9514626ad3cf762e330952b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4] to archive 2023-03-24 01:54:50,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:54:50,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59b785473b1f413fa0c62b2ea2cf2a69 2023-03-24 01:54:50,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/cd8f091d7f514f13948cb962c2a309ff 2023-03-24 01:54:50,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a12f54fa0a464beabb5053d580857768 2023-03-24 01:54:50,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a4864c72035446c493d0d1e9582efa37 2023-03-24 01:54:50,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/9a61ce5c9557436583806f0bcfd5f1d2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/9a61ce5c9557436583806f0bcfd5f1d2 2023-03-24 01:54:50,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/5324ee6a031a412682154a84dd12c2a8 2023-03-24 01:54:50,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57ff0f5a56db48999a10063d31ac355f 2023-03-24 01:54:50,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/32d8e1fe0468433183339123639bb29c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/32d8e1fe0468433183339123639bb29c 2023-03-24 01:54:50,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b651ddd41e3e4d94af56bcaba38595f8 2023-03-24 01:54:50,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/07126a8badd34668a5eb89fbb1233e31 2023-03-24 01:54:50,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a04853af62994df7a04b65c3547f04ec to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/a04853af62994df7a04b65c3547f04ec 2023-03-24 01:54:50,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ea6e4898ef0d47a88881973c3bea6d7e 2023-03-24 01:54:50,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ae28e0b6a2d64df0b723fa730ff07991 2023-03-24 01:54:50,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/7302f5762c854508a4ea9925f755e8e9 2023-03-24 01:54:50,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/05be51a713524919ab447e710736fddc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/05be51a713524919ab447e710736fddc 2023-03-24 01:54:50,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6b86e4fdfe744cffaff114ab55cf90d3 2023-03-24 01:54:50,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/3c00c795072f4a509e413ee7c97b85c1 2023-03-24 01:54:50,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59fd29387b8d423b8a3805abc503ea0f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/59fd29387b8d423b8a3805abc503ea0f 2023-03-24 01:54:50,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/f9ef1b9a90234bdd805c88940c92be66 2023-03-24 01:54:50,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/708c182d0a524672a4de44f49d400314 2023-03-24 01:54:50,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/6f0f87bf4266401c8522c3074bedea6e 2023-03-24 01:54:50,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/2bba103436d649c594c93d8176523b7a 2023-03-24 01:54:50,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57fd90d6a9514626ad3cf762e330952b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/57fd90d6a9514626ad3cf762e330952b 2023-03-24 01:54:50,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ddc1af4dbb7144dd96f1df57bb429944 2023-03-24 01:54:50,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/61de23f7301b4dde9a7ba9b1065574bb 2023-03-24 01:54:50,294 DEBUG [StoreCloser-TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/b9f43086495e4b75b797dc895cd024f4 2023-03-24 01:54:50,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits/474.seqid, newMaxSeqId=474, maxSeqId=4 2023-03-24 01:54:50,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7. 2023-03-24 01:54:50,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 38628ffe26cfdb42701ddddf336a55f7: 2023-03-24 01:54:50,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:50,300 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=38628ffe26cfdb42701ddddf336a55f7, regionState=CLOSED 2023-03-24 01:54:50,301 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622890300"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622890300"}]},"ts":"1679622890300"} 2023-03-24 01:54:50,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-24 01:54:50,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 38628ffe26cfdb42701ddddf336a55f7, server=jenkins-hbase4.apache.org,44439,1679622817494 in 1.9120 sec 2023-03-24 01:54:50,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-24 01:54:50,304 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=38628ffe26cfdb42701ddddf336a55f7, UNASSIGN in 1.9170 sec 2023-03-24 01:54:50,305 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622890304"}]},"ts":"1679622890304"} 2023-03-24 01:54:50,305 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:54:50,307 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:54:50,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9320 sec 2023-03-24 01:54:50,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 01:54:50,483 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-24 01:54:50,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:54:50,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,485 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 01:54:50,486 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,491 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:50,492 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits] 2023-03-24 01:54:50,495 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6399b243cca54e48806797ca9596f0d9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/6399b243cca54e48806797ca9596f0d9 2023-03-24 01:54:50,496 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/b5166c553ba04d2a93dcc6ad166fcf12 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/A/b5166c553ba04d2a93dcc6ad166fcf12 2023-03-24 01:54:50,498 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/57256a6d093f42058f0d428d45a9e083 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/57256a6d093f42058f0d428d45a9e083 2023-03-24 01:54:50,499 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5b52e574878b42c3b7943fe7b97343a8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/B/5b52e574878b42c3b7943fe7b97343a8 2023-03-24 01:54:50,501 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/1430ebf26fe64cb99e9c82e8072a88b0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/1430ebf26fe64cb99e9c82e8072a88b0 2023-03-24 01:54:50,502 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ce697f8271cf4888b8243c82e2286020 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/C/ce697f8271cf4888b8243c82e2286020 2023-03-24 01:54:50,504 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits/474.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7/recovered.edits/474.seqid 2023-03-24 01:54:50,505 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/38628ffe26cfdb42701ddddf336a55f7 2023-03-24 01:54:50,505 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:54:50,505 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:54:50,506 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 01:54:50,509 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324042361ab86d9408da609ead36fca78f3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324042361ab86d9408da609ead36fca78f3 2023-03-24 01:54:50,510 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032408f41380244048ab818e5727f201d29f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032408f41380244048ab818e5727f201d29f 2023-03-24 01:54:50,511 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc5f0ac48524361a5c9ed0eacb3bacf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc5f0ac48524361a5c9ed0eacb3bacf 2023-03-24 01:54:50,512 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240fcad6580bb04345980ebdc3dd0c4437 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240fcad6580bb04345980ebdc3dd0c4437 2023-03-24 01:54:50,513 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241790d2bd84914d47af21e3bfdabef7e1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241790d2bd84914d47af21e3bfdabef7e1 2023-03-24 01:54:50,514 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242d1a4554e35748449d7b10fd7f268d18 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242d1a4554e35748449d7b10fd7f268d18 2023-03-24 01:54:50,514 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245ac5294591de465b882c596f24c8bd7d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245ac5294591de465b882c596f24c8bd7d 2023-03-24 01:54:50,515 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324603ed262a7e548b581c1b2327739959d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324603ed262a7e548b581c1b2327739959d 2023-03-24 01:54:50,516 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324627f0ed605a14e61b287a88178ecc5e5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324627f0ed605a14e61b287a88178ecc5e5 2023-03-24 01:54:50,517 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324717e384337df4caf90006d471952a358 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324717e384337df4caf90006d471952a358 2023-03-24 01:54:50,518 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471ecefcffe7445ae9aa89dd2e5ebb0a6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471ecefcffe7445ae9aa89dd2e5ebb0a6 2023-03-24 01:54:50,519 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b18dd0018bb4466bbe3d0e4c35e797a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b18dd0018bb4466bbe3d0e4c35e797a 2023-03-24 01:54:50,520 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032480ffd40e8d88409e83167c219218b32c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032480ffd40e8d88409e83167c219218b32c 2023-03-24 01:54:50,521 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248572a559dff6445c87123d5b7dab3bd4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248572a559dff6445c87123d5b7dab3bd4 2023-03-24 01:54:50,522 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324859dea6a8a304b559e965b0352b5d909 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324859dea6a8a304b559e965b0352b5d909 2023-03-24 01:54:50,523 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324892fe7c05a2148b9a61f0b0048eec29c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324892fe7c05a2148b9a61f0b0048eec29c 2023-03-24 01:54:50,524 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ab6dd158decb4478bd8b422c56ad8c27 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ab6dd158decb4478bd8b422c56ad8c27 2023-03-24 01:54:50,525 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cde1ea98fdf44b2cb90117686679bad1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cde1ea98fdf44b2cb90117686679bad1 2023-03-24 01:54:50,526 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e599a7d419f4459083f0a1b30e5bbfdc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e599a7d419f4459083f0a1b30e5bbfdc 2023-03-24 01:54:50,527 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f8120a9f8d8e4f19a4fc5d5cde03df5a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f8120a9f8d8e4f19a4fc5d5cde03df5a 2023-03-24 01:54:50,528 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f84547b389e94da6ab0271dd707ac2c9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f84547b389e94da6ab0271dd707ac2c9 2023-03-24 01:54:50,529 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:54:50,531 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,533 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:54:50,534 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:54:50,535 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,535 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:54:50,535 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679622890535"}]},"ts":"9223372036854775807"} 2023-03-24 01:54:50,537 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:54:50,537 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 38628ffe26cfdb42701ddddf336a55f7, NAME => 'TestAcidGuarantees,,1679622854360.38628ffe26cfdb42701ddddf336a55f7.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:54:50,537 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:54:50,537 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679622890537"}]},"ts":"9223372036854775807"} 2023-03-24 01:54:50,538 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:54:50,540 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,541 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-24 01:54:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 01:54:50,587 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-24 01:54:50,599 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 231), OpenFileDescriptor=523 (was 557), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=283 (was 296), ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=5596 (was 5834) 2023-03-24 01:54:50,610 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=523, MaxFileDescriptor=60000, SystemLoadAverage=283, ProcessCount=172, AvailableMemoryMB=5596 2023-03-24 01:54:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:54:50,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:54:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:54:50,614 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:54:50,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-24 01:54:50,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:54:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:50,616 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:50,617 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a empty. 2023-03-24 01:54:50,617 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:50,617 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:54:50,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:51,024 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:54:51,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d337c30c8fcf8ea343af6ac65f0aa65a, NAME => 'TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:54:51,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d337c30c8fcf8ea343af6ac65f0aa65a, disabling compactions & flushes 2023-03-24 01:54:51,432 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. after waiting 0 ms 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,432 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,432 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:51,434 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:54:51,435 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622891435"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622891435"}]},"ts":"1679622891435"} 2023-03-24 01:54:51,436 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:54:51,437 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:54:51,437 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622891437"}]},"ts":"1679622891437"} 2023-03-24 01:54:51,438 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:54:51,441 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, ASSIGN}] 2023-03-24 01:54:51,442 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, ASSIGN 2023-03-24 01:54:51,443 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:54:51,594 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d337c30c8fcf8ea343af6ac65f0aa65a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:51,594 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622891594"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622891594"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622891594"}]},"ts":"1679622891594"} 2023-03-24 01:54:51,596 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:54:51,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:51,750 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,750 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d337c30c8fcf8ea343af6ac65f0aa65a, NAME => 'TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:54:51,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:54:51,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,751 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,752 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,753 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A 2023-03-24 01:54:51,753 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A 2023-03-24 01:54:51,753 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:51,754 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d337c30c8fcf8ea343af6ac65f0aa65a columnFamilyName A 2023-03-24 01:54:51,754 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(310): Store=d337c30c8fcf8ea343af6ac65f0aa65a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:51,754 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,755 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B 2023-03-24 01:54:51,755 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B 2023-03-24 01:54:51,755 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:51,756 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d337c30c8fcf8ea343af6ac65f0aa65a columnFamilyName B 2023-03-24 01:54:51,756 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(310): Store=d337c30c8fcf8ea343af6ac65f0aa65a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:51,756 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,757 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C 2023-03-24 01:54:51,757 DEBUG [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C 2023-03-24 01:54:51,757 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:54:51,757 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d337c30c8fcf8ea343af6ac65f0aa65a columnFamilyName C 2023-03-24 01:54:51,758 INFO [StoreOpener-d337c30c8fcf8ea343af6ac65f0aa65a-1] regionserver.HStore(310): Store=d337c30c8fcf8ea343af6ac65f0aa65a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:54:51,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,758 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,759 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,761 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:54:51,762 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:51,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:54:51,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d337c30c8fcf8ea343af6ac65f0aa65a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60951432, jitterRate=-0.09175288677215576}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:54:51,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:51,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., pid=30, masterSystemTime=1679622891747 2023-03-24 01:54:51,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:51,766 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d337c30c8fcf8ea343af6ac65f0aa65a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:51,767 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622891766"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622891766"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622891766"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622891766"}]},"ts":"1679622891766"} 2023-03-24 01:54:51,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-24 01:54:51,769 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 in 172 msec 2023-03-24 01:54:51,770 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-24 01:54:51,770 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, ASSIGN in 328 msec 2023-03-24 01:54:51,770 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:54:51,771 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622891770"}]},"ts":"1679622891770"} 2023-03-24 01:54:51,771 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:54:51,773 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:54:51,774 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-03-24 01:54:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 01:54:52,720 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-24 01:54:52,721 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x7233cccd to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,726 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@112921f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,727 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,728 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,729 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:54:52,731 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48078, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:54:52,732 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x12e82edd to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,736 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed1dde9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,737 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,742 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,742 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,746 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,747 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,753 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,753 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,757 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,758 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x7810a231 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,765 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@104dbf4e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,765 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,770 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,771 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,775 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,776 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,780 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,781 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:54:52,785 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:54:52,789 DEBUG [hconnection-0x7c78163c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,789 DEBUG [hconnection-0x423608be-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,790 DEBUG [hconnection-0x11eb8477-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,790 DEBUG [hconnection-0x2fa75ab9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,790 DEBUG [hconnection-0x7c6c3b2d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,791 DEBUG [hconnection-0x2b70e3e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,792 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34892, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,792 DEBUG [hconnection-0x510aa473-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:52,792 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34912, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,792 DEBUG [hconnection-0xf85e176-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,793 DEBUG [hconnection-0x52ebddbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,797 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,800 DEBUG [hconnection-0x472e01b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:54:52,803 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34868, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,803 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34880, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,804 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34906, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,804 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34910, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,804 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34920, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,804 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,807 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34948, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:52,809 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:52,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:52,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:52,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:52,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:52,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:52,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:52,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:52,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:52,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:52,825 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:52,825 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:52,825 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:52,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:52,828 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:52,830 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:52,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622892847, max timestamp=1679622892853Num uniques 859; 2023-03-24 01:54:52,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:52,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622892847, max timestamp=1679622892853Num uniques 859; 2023-03-24 01:54:52,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:52,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622892847, max timestamp=1679622892853Num uniques 859; 2023-03-24 01:54:52,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:54:52,855 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/84f52e4e459644afa93f5fb654f0f576 2023-03-24 01:54:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622952876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622952878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622952884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622952885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622952885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622952986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622952986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622952989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622952990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:52,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:52,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622952990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622953190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622953191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622953193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622953194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622953195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,307 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/0d1bfaa4fbaf489a99a2a3f36fac11c2 2023-03-24 01:54:53,338 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6e07f69df56b4c74b5d24d9aa0626313 2023-03-24 01:54:53,343 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/84f52e4e459644afa93f5fb654f0f576 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576 2023-03-24 01:54:53,347 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 01:54:53,349 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/0d1bfaa4fbaf489a99a2a3f36fac11c2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2 2023-03-24 01:54:53,362 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 01:54:53,366 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6e07f69df56b4c74b5d24d9aa0626313 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313 2023-03-24 01:54:53,377 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 01:54:53,378 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d337c30c8fcf8ea343af6ac65f0aa65a in 550ms, sequenceid=12, compaction requested=false 2023-03-24 01:54:53,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:53,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:53,386 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:53,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b558613[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:53,388 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:53,388 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,389 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:53,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:53,390 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,390 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:53,395 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,395 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,396 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:53,399 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:53,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:53,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:53,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:53,399 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:53,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,403 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,407 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:53,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:53,407 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:53,408 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:53,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:53,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-24 01:54:53,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622892865, max timestamp=1679622893496Num uniques 1068; 2023-03-24 01:54:53,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 01:54:53,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622892865, max timestamp=1679622893496Num uniques 1068; 2023-03-24 01:54:53,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622892865, max timestamp=1679622893496Num uniques 1068; 2023-03-24 01:54:53,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 01:54:53,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 01:54:53,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:53,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:53,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:53,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,508 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:53,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:53,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/59ea8e08df4f40c48298c3b3c7850747 2023-03-24 01:54:53,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:53,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622893510, max timestamp=1679622893514Num uniques 859; 2023-03-24 01:54:53,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:53,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622893510, max timestamp=1679622893515Num uniques 859; 2023-03-24 01:54:53,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:53,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622893510, max timestamp=1679622893514Num uniques 859; 2023-03-24 01:54:53,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:53,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:53,530 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:53,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:53,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:53,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/396fa83616ee465c96602d3fd3f7a61d 2023-03-24 01:54:53,533 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,533 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:53,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622893526, max timestamp=1679622893533Num uniques 1018; 2023-03-24 01:54:53,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:53,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,537 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:53,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622893526, max timestamp=1679622893533Num uniques 1018; 2023-03-24 01:54:53,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:54:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:53,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622893526, max timestamp=1679622893533Num uniques 1018; 2023-03-24 01:54:53,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 01:54:53,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:53,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:53,540 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:53,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/d4a77986dd71467e904a0330213ed4e0 2023-03-24 01:54:53,552 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,552 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:53,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:53,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14182ffa[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:53,556 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:53,556 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:53,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:53,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/59ea8e08df4f40c48298c3b3c7850747 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747 2023-03-24 01:54:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:53,564 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:53,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:53,564 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:53,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747, entries=200, sequenceid=50, filesize=13.9 K 2023-03-24 01:54:53,566 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,566 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/396fa83616ee465c96602d3fd3f7a61d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d 2023-03-24 01:54:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679622893542, max timestamp=1679622893563Num uniques 1027; 2023-03-24 01:54:53,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 01:54:53,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679622893542, max timestamp=1679622893563Num uniques 1027; 2023-03-24 01:54:53,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 01:54:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679622893542, max timestamp=1679622893563Num uniques 1009; 2023-03-24 01:54:53,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-03-24 01:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 01:54:53,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/d4a77986dd71467e904a0330213ed4e0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:53,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:53,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:53,579 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:53,579 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:53,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 01:54:53,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=73.80 KB/75570 for d337c30c8fcf8ea343af6ac65f0aa65a in 100ms, sequenceid=50, compaction requested=false 2023-03-24 01:54:53,596 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:53,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:53,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=87.22 KB heapSize=239.16 KB 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:53,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/6cd9adc4e8934ce3b0bd48496c57dc3c 2023-03-24 01:54:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622953608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622953609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622953610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622953612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622953613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3b3125186144bc6834a8b999066b499 2023-03-24 01:54:53,679 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:53,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:53,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:53,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:53,682 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:53,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:53,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:53,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,685 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:53,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:53,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:53,686 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,686 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:53,687 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:53,687 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:53,687 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:53,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:53,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1f5725[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:53,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:53,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,695 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:53,695 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,695 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:53,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:53,700 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:53,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:53,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:53,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:53,700 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:53,702 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,702 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,702 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:53,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:53,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,705 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:53,706 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:53,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,706 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:53,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622953716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622953716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622953716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622953717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622953718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,806 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:53,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:53,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:53,810 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:53,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:53,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:53,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:53,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:53,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:53,813 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:53,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:53,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:53,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:53,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc160e5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:53,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:53,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:53,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:53,820 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:53,823 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:53,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:53,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:53,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:53,823 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:53,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:53,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:53,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:53,829 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:53,830 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:53,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622953918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622953919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622953920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622953920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:53,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622953921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,930 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:53,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:53,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:53,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:53,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:53,933 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:53,933 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:53,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:53,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,935 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:53,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:53,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:53,937 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,937 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:53,937 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:53,937 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:53,937 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:53,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:53,942 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:53,942 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c930090[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:53,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:53,948 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:53,950 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:53,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:53,954 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:53,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:53,954 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:53,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:53,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:53,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:53,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:53,961 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:53,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:53,962 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:53,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/5ce01d1ac67d416a97adf42cb8da9c6a 2023-03-24 01:54:54,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/6cd9adc4e8934ce3b0bd48496c57dc3c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c 2023-03-24 01:54:54,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c, entries=150, sequenceid=114, filesize=11.6 K 2023-03-24 01:54:54,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3b3125186144bc6834a8b999066b499 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499 2023-03-24 01:54:54,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499, entries=150, sequenceid=114, filesize=11.6 K 2023-03-24 01:54:54,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/5ce01d1ac67d416a97adf42cb8da9c6a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a 2023-03-24 01:54:54,062 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a, entries=150, sequenceid=114, filesize=11.6 K 2023-03-24 01:54:54,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~259.03 KB/265248, currentSize=93.93 KB/96180 for d337c30c8fcf8ea343af6ac65f0aa65a in 468ms, sequenceid=114, compaction requested=true 2023-03-24 01:54:54,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:54,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:54,065 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:54,065 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:54,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:54,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:54,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,068 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:54,068 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:54:54,068 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,068 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=37.1 K 2023-03-24 01:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,069 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,072 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:54,072 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 84f52e4e459644afa93f5fb654f0f576, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679622892809 2023-03-24 01:54:54,072 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:54:54,072 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,072 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=34.8 K 2023-03-24 01:54:54,074 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0d1bfaa4fbaf489a99a2a3f36fac11c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679622892809 2023-03-24 01:54:54,074 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 59ea8e08df4f40c48298c3b3c7850747, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622892865 2023-03-24 01:54:54,075 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 396fa83616ee465c96602d3fd3f7a61d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622892865 2023-03-24 01:54:54,075 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6cd9adc4e8934ce3b0bd48496c57dc3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:54,075 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e3b3125186144bc6834a8b999066b499, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:54,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,080 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:54:54,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:54,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,081 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:54,081 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,081 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:54,081 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,083 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:54,084 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#199 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:54,107 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/4e11dc8e0ae4425d9ffd17d661327156 2023-03-24 01:54:54,117 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/180e30fc039445109ddfd97835fb180b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/180e30fc039445109ddfd97835fb180b 2023-03-24 01:54:54,121 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 180e30fc039445109ddfd97835fb180b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:54,122 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:54,122 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622894065; duration=0sec 2023-03-24 01:54:54,122 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:54,122 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:54,123 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:54,123 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/8b613a77905f40e8a69af75cfc7a973e 2023-03-24 01:54:54,123 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:54:54,123 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,123 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=34.8 K 2023-03-24 01:54:54,124 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6e07f69df56b4c74b5d24d9aa0626313, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679622892809 2023-03-24 01:54:54,124 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d4a77986dd71467e904a0330213ed4e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679622892865 2023-03-24 01:54:54,124 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5ce01d1ac67d416a97adf42cb8da9c6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:54,130 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#203 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:54,140 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/ad50519edb374363be6a6e4c5af6203d 2023-03-24 01:54:54,146 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/4e11dc8e0ae4425d9ffd17d661327156 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156 2023-03-24 01:54:54,150 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6c74a1de664c438baffba066872a3ecb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6c74a1de664c438baffba066872a3ecb 2023-03-24 01:54:54,151 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156, entries=150, sequenceid=131, filesize=11.7 K 2023-03-24 01:54:54,156 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/8b613a77905f40e8a69af75cfc7a973e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e 2023-03-24 01:54:54,159 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 6c74a1de664c438baffba066872a3ecb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:54,159 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:54,159 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622894066; duration=0sec 2023-03-24 01:54:54,159 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:54,168 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e, entries=150, sequenceid=131, filesize=11.7 K 2023-03-24 01:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/ad50519edb374363be6a6e4c5af6203d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d 2023-03-24 01:54:54,173 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d, entries=150, sequenceid=131, filesize=11.7 K 2023-03-24 01:54:54,173 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for d337c30c8fcf8ea343af6ac65f0aa65a in 93ms, sequenceid=131, compaction requested=false 2023-03-24 01:54:54,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:54,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,175 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a18507[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,179 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,184 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,187 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,187 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,194 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,194 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:54,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:54,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:54,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622894248, max timestamp=1679622894253Num uniques 859; 2023-03-24 01:54:54,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:54,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622894248, max timestamp=1679622894253Num uniques 859; 2023-03-24 01:54:54,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622894248, max timestamp=1679622894253Num uniques 859; 2023-03-24 01:54:54,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:54,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:54:54,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622954266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622954267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622954268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622954269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622954271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,294 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,297 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,299 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,301 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,301 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,301 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,301 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,301 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,305 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c624b69[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,307 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,307 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,307 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,311 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,311 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,319 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:54,319 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622954373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622954373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622954373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622954374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622954375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,420 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,423 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,429 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c74de48[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,456 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,456 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:54,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,468 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,468 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,505 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/eeb25520992341bc97ce4bbb0789b8b1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/eeb25520992341bc97ce4bbb0789b8b1 2023-03-24 01:54:54,509 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into eeb25520992341bc97ce4bbb0789b8b1(size=11.7 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:54,509 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:54,509 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622894065; duration=0sec 2023-03-24 01:54:54,509 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:54,568 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,571 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,571 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,573 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,574 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,575 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7897c8d6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622954576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622954576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622954577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622954577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622954579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,586 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,590 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,595 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,595 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/5a0569e8c95a42a0a2ddb01a57bd02f6 2023-03-24 01:54:54,695 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,699 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,701 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,702 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,702 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,702 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,703 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dabf37e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,707 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,712 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,713 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,718 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,719 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,719 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,819 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,823 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,824 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,825 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,829 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,829 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,829 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,829 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,832 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35cd0a1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,838 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,838 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,838 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,843 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,843 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,845 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,845 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,845 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,851 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,852 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622954880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622954880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622954881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622954881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622954882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,952 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:54,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:54,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:54,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:54,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:54,955 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:54,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:54,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:54,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,957 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:54,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:54,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:54,959 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,959 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:54,959 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:54,959 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:54,960 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:54,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:54,962 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2970c81c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:54,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:54,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:54,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:54,969 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:54,971 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:54,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:54,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:54,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:54,971 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:54,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:54,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,973 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:54,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:54,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:54,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:54,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,977 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:54,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:54,977 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:54,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:54,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:54,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/61d1444ab0c54f55a738085a10161b8f 2023-03-24 01:54:55,078 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:55,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:55,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:55,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:55,081 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:55,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:55,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:55,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:55,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:55,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f67749c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:55,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:55,089 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:55,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:55,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:55,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:55,094 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:55,094 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:55,094 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:55,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:55,095 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:55,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,102 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,102 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:55,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,102 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:55,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,202 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:55,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:55,208 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:55,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:55,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:55,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:55,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:55,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:55,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:55,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:55,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc54a0d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:55,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:55,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:55,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:55,226 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:55,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:55,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:55,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:55,226 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:55,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:55,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:55,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,232 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:55,232 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:55,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,333 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:55,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:55,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:55,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:55,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:55,336 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:55,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:55,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:55,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:55,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:55,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:55,341 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,341 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:55,341 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:55,341 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:55,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:55,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:55,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18f954cc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:55,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:55,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:55,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:55,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:55,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:55,350 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:55,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:55,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:55,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:55,350 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:55,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:55,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:55,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:55,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,360 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:55,360 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622955385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622955386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622955386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622955386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622955389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,460 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:55,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:55,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:55,463 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:55,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:55,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:55,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,465 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:55,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:55,467 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,467 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:55,467 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:55,467 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:55,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:55,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:55,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:55,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e64fe7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:55,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:55,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:55,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:55,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:55,476 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:55,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:55,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:55,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:55,476 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:55,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:55,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:55,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:55,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:55,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:55,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:55,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,481 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:55,481 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:55,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/58fd531cabaa4cc9aa553f9bbc33a702 2023-03-24 01:54:55,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/5a0569e8c95a42a0a2ddb01a57bd02f6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6 2023-03-24 01:54:55,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6, entries=150, sequenceid=144, filesize=11.8 K 2023-03-24 01:54:55,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/61d1444ab0c54f55a738085a10161b8f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f 2023-03-24 01:54:55,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f, entries=150, sequenceid=144, filesize=11.8 K 2023-03-24 01:54:55,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/58fd531cabaa4cc9aa553f9bbc33a702 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702 2023-03-24 01:54:55,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702, entries=150, sequenceid=144, filesize=11.8 K 2023-03-24 01:54:55,503 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d337c30c8fcf8ea343af6ac65f0aa65a in 1275ms, sequenceid=144, compaction requested=true 2023-03-24 01:54:55,503 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:55,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:55,503 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:55,503 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:55,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:55,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:55,506 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36011 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:55,506 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:54:55,506 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36011 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:55,506 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,506 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:54:55,506 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/180e30fc039445109ddfd97835fb180b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=35.2 K 2023-03-24 01:54:55,506 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,507 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/eeb25520992341bc97ce4bbb0789b8b1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=35.2 K 2023-03-24 01:54:55,507 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting eeb25520992341bc97ce4bbb0789b8b1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:55,514 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8b613a77905f40e8a69af75cfc7a973e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1679622893608 2023-03-24 01:54:55,514 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 180e30fc039445109ddfd97835fb180b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:55,514 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 61d1444ab0c54f55a738085a10161b8f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:55,514 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4e11dc8e0ae4425d9ffd17d661327156, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1679622893608 2023-03-24 01:54:55,515 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5a0569e8c95a42a0a2ddb01a57bd02f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:55,521 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:55,521 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:55,543 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/4e79ca3133134ea794549e3e74183481 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e79ca3133134ea794549e3e74183481 2023-03-24 01:54:55,548 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 4e79ca3133134ea794549e3e74183481(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:55,548 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:55,548 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622895503; duration=0sec 2023-03-24 01:54:55,548 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:55,548 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:55,549 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36011 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:55,549 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:54:55,550 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,550 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6c74a1de664c438baffba066872a3ecb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=35.2 K 2023-03-24 01:54:55,550 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6c74a1de664c438baffba066872a3ecb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1679622893585 2023-03-24 01:54:55,550 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ad50519edb374363be6a6e4c5af6203d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1679622893608 2023-03-24 01:54:55,551 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 58fd531cabaa4cc9aa553f9bbc33a702, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:55,556 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:55,582 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:55,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:55,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:55,585 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:55,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:55,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:55,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:55,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:55,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:55,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:55,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:55,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:55,592 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:55,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:54:55,833 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:54:55,944 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/8d71fd8e092a4642a1bd32c331202be3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8d71fd8e092a4642a1bd32c331202be3 2023-03-24 01:54:55,949 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 8d71fd8e092a4642a1bd32c331202be3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:55,949 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:55,949 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622895503; duration=0sec 2023-03-24 01:54:55,949 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:55,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/3a803cd2db4542b3a5a9e7e315d3bf86 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/3a803cd2db4542b3a5a9e7e315d3bf86 2023-03-24 01:54:55,973 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 3a803cd2db4542b3a5a9e7e315d3bf86(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:55,973 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:55,973 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622895503; duration=0sec 2023-03-24 01:54:55,973 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:56,025 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d5f325fc86cb4fdf8612dd02b7e38683 2023-03-24 01:54:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:56,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:56,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622956399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622956399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622956402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622956402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622956412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,455 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3c50504a49e4444bd8e62efe40d8580 2023-03-24 01:54:56,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622956503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622956503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622956505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622956505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622956515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622956707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622956707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622956708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622956709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:56,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622956718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,870 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6a3a5c1c1a38464e96cde7e1d52aca95 2023-03-24 01:54:56,874 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d5f325fc86cb4fdf8612dd02b7e38683 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683 2023-03-24 01:54:56,878 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 01:54:56,879 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3c50504a49e4444bd8e62efe40d8580 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580 2023-03-24 01:54:56,885 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 01:54:56,886 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6a3a5c1c1a38464e96cde7e1d52aca95 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95 2023-03-24 01:54:56,891 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95, entries=150, sequenceid=182, filesize=11.8 K 2023-03-24 01:54:56,891 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for d337c30c8fcf8ea343af6ac65f0aa65a in 1299ms, sequenceid=182, compaction requested=false 2023-03-24 01:54:56,891 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:56,892 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:56,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:56,893 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63711e13[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:56,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:56,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,897 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:56,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:56,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:56,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:56,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:56,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:56,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:56,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:56,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:56,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:56,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:56,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,904 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:56,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,905 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:56,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:56,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:56,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:56,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:56,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:56,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:56,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:56,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:56,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:56,909 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:56,909 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,009 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:57,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:57,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,014 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,014 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,014 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,014 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,014 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,014 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,016 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,016 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,023 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,023 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,024 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,024 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,026 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622957031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622957033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622957034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622957034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622957035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3433a552[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,041 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,041 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,043 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,045 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,045 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,047 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,048 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,048 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,063 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,063 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622957136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622957137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622957139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622957139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622957140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,163 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,166 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,166 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,167 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,167 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,168 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,170 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,170 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,170 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,170 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,170 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,171 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bed82c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,174 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,177 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,179 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,181 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,181 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,181 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,181 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,185 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,185 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,186 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,186 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,286 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,289 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,293 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,293 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,293 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,293 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,296 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63223a6b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,297 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,298 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,299 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,303 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,303 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,304 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,309 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,309 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,310 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:57,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622957339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622957339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622957343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622957343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622957344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,410 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,413 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,417 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,419 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,419 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@848319a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,421 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,421 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,423 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,424 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,424 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,426 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,429 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,430 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,431 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/5a638f19c1dd477cb3d8f700eae721e4 2023-03-24 01:54:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,433 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,437 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,438 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,438 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,538 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,542 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,548 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,548 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,548 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,550 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,551 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,551 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,557 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21780cf4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,557 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,558 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,558 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,559 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,562 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,562 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,564 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,564 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,564 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,565 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,569 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,569 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622957641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622957643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622957645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622957645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622957647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,670 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,673 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,675 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,677 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,677 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,677 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,677 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,677 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3197df4f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,681 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,683 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,683 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,684 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,687 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,687 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,692 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,692 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:57,693 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,693 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,793 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,796 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,797 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,797 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,799 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,799 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,799 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,800 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,800 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,800 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:57,800 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:57,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:57,802 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:57,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43144a74[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:57,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:57,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:57,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:57,806 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:57,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:57,808 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:57,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:57,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:57,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:57,808 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:57,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,810 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:57,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:57,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,814 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:57,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:57,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:57,814 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:57,815 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:57,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,815 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:57,815 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32 2023-03-24 01:54:57,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/4bf18bc9066b4aafb678856ae20c2eea 2023-03-24 01:54:57,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/5a638f19c1dd477cb3d8f700eae721e4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4 2023-03-24 01:54:57,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4, entries=200, sequenceid=199, filesize=14.1 K 2023-03-24 01:54:57,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32 2023-03-24 01:54:57,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32, entries=150, sequenceid=199, filesize=11.8 K 2023-03-24 01:54:57,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/4bf18bc9066b4aafb678856ae20c2eea as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea 2023-03-24 01:54:57,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea, entries=150, sequenceid=199, filesize=11.8 K 2023-03-24 01:54:57,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for d337c30c8fcf8ea343af6ac65f0aa65a in 870ms, sequenceid=199, compaction requested=true 2023-03-24 01:54:57,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:57,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:57,882 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:57,882 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:57,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:57,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:57,883 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:57,883 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:57,883 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:54:57,883 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:54:57,884 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,884 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,884 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e79ca3133134ea794549e3e74183481, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=35.5 K 2023-03-24 01:54:57,884 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8d71fd8e092a4642a1bd32c331202be3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=37.8 K 2023-03-24 01:54:57,884 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4e79ca3133134ea794549e3e74183481, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:57,884 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8d71fd8e092a4642a1bd32c331202be3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:57,885 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e3c50504a49e4444bd8e62efe40d8580, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679622894266 2023-03-24 01:54:57,885 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d5f325fc86cb4fdf8612dd02b7e38683, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679622894266 2023-03-24 01:54:57,885 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a6a64f6d7ee44d5ebacc7e87e6e1aa32, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:54:57,885 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5a638f19c1dd477cb3d8f700eae721e4, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:54:57,897 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:57,901 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:57,915 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:57,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:57,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:57,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:57,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:57,920 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:57,920 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:57,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:57,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,921 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:57,922 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:57,923 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,923 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:57,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:57,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:57,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:57,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:57,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:57,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:57,925 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,925 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:57,925 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:54:57,925 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:57,926 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,926 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:57,926 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,926 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:57,926 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:57,928 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/293e24c2c5d34ee2bd4275b2a8526d6d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/293e24c2c5d34ee2bd4275b2a8526d6d 2023-03-24 01:54:57,934 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 293e24c2c5d34ee2bd4275b2a8526d6d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:57,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:57,934 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622897882; duration=0sec 2023-03-24 01:54:57,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:54:57,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:54:57,936 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:54:57,936 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:54:57,936 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:57,936 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/3a803cd2db4542b3a5a9e7e315d3bf86, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=35.5 K 2023-03-24 01:54:57,936 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3a803cd2db4542b3a5a9e7e315d3bf86, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679622894225 2023-03-24 01:54:57,936 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6a3a5c1c1a38464e96cde7e1d52aca95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679622894266 2023-03-24 01:54:57,937 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4bf18bc9066b4aafb678856ae20c2eea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:54:57,967 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#219 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:54:57,970 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/20f3a705c06844f1b572ad1703587428 2023-03-24 01:54:58,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:58,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:58,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622958156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622958156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622958159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622958160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622958161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622958261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622958261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622958263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622958264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622958264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,319 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/1ba1b199dc29400e9cc8768342c699c6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1ba1b199dc29400e9cc8768342c699c6 2023-03-24 01:54:58,324 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 1ba1b199dc29400e9cc8768342c699c6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:58,324 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:58,324 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622897882; duration=0sec 2023-03-24 01:54:58,325 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:58,388 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/6234e2a489654728a5d00e2a19dcc29b 2023-03-24 01:54:58,389 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/234ee3adc208446cbaaef33dfab9748c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/234ee3adc208446cbaaef33dfab9748c 2023-03-24 01:54:58,403 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 234ee3adc208446cbaaef33dfab9748c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:54:58,403 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:58,403 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622897882; duration=0sec 2023-03-24 01:54:58,403 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:54:58,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622958465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622958466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622958467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622958469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622958470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622958767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622958769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622958770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622958772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:58,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622958772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,822 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/c5854221a55248b791623bdaeb3bcf1a 2023-03-24 01:54:58,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/20f3a705c06844f1b572ad1703587428 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428 2023-03-24 01:54:58,830 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428, entries=150, sequenceid=217, filesize=11.8 K 2023-03-24 01:54:58,831 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/6234e2a489654728a5d00e2a19dcc29b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b 2023-03-24 01:54:58,835 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b, entries=150, sequenceid=217, filesize=11.8 K 2023-03-24 01:54:58,835 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/c5854221a55248b791623bdaeb3bcf1a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a 2023-03-24 01:54:58,839 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a, entries=150, sequenceid=217, filesize=11.8 K 2023-03-24 01:54:58,840 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for d337c30c8fcf8ea343af6ac65f0aa65a in 915ms, sequenceid=217, compaction requested=false 2023-03-24 01:54:58,840 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:58,840 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:58,840 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:58,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:58,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a19676d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:58,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:58,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:58,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:58,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:58,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:58,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:58,851 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:58,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:58,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:58,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:58,851 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:58,853 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,853 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:58,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,853 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:58,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:58,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:58,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,855 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:58,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:58,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:58,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:58,858 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:58,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:58,858 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:58,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:58,958 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:58,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:58,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:58,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:58,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:58,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:58,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:58,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:58,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:58,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:58,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:58,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:58,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:58,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:58,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:58,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:58,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:58,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:58,966 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:58,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:58,973 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/dbecaece114249f7a2f3bb6c3524e484 2023-03-24 01:54:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:59,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:59,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622959283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622959283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622959283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622959287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622959287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,383 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/aa94c84733fc413180744793b76b2c0b 2023-03-24 01:54:59,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622959388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622959389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622959389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622959391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,393 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/5ba0866736924f62b5f54faa1bbc8e7a 2023-03-24 01:54:59,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622959391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,396 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/dbecaece114249f7a2f3bb6c3524e484 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484 2023-03-24 01:54:59,399 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 01:54:59,400 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/aa94c84733fc413180744793b76b2c0b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b 2023-03-24 01:54:59,404 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 01:54:59,405 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/5ba0866736924f62b5f54faa1bbc8e7a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a 2023-03-24 01:54:59,408 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 01:54:59,409 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d337c30c8fcf8ea343af6ac65f0aa65a in 443ms, sequenceid=235, compaction requested=true 2023-03-24 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:54:59,409 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:54:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:54:59,411 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:54:59,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b67c576[Count = 0] remaining members to acquire global barrier 2023-03-24 01:54:59,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:54:59,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,414 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:54:59,414 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:54:59,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:54:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:54:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:54:59,416 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:54:59,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:54:59,419 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:54:59,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:54:59,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:54:59,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:54:59,419 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:54:59,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:59,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:54:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:54:59,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:59,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:54:59,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:54:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:54:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:54:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:59,425 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:54:59,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:54:59,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:54:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:59,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:54:59,426 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:54:59,426 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:54:59,426 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:54:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:54:59,426 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,426 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:54:59,426 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,526 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:54:59,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:54:59,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:54:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:54:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:54:59,529 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:59,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:54:59,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:54:59,532 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:59,532 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:54:59,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:54:59,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:54:59,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:54:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:54:59,533 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:54:59,533 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:54:59,533 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:54:59,533 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:54:59,534 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:54:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:54:59,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:54:59,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622959599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622959600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622959601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622959602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622959602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622959703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622959704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622959704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622959705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622959706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622959906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622959908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622959908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622959908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:54:59,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622959909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:54:59,941 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/f340cb89eb82421c84fb8232f736e25b 2023-03-24 01:55:00,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622960208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622960211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622960211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622960213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622960213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,353 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e9f6045c839a421db18a8ccaf7ac24e4 2023-03-24 01:55:00,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622960711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622960715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622960715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622960718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:00,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622960719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,763 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/bd43257641204d44b0c9fa8d8c6388ff 2023-03-24 01:55:00,767 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/f340cb89eb82421c84fb8232f736e25b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b 2023-03-24 01:55:00,770 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b, entries=150, sequenceid=252, filesize=11.8 K 2023-03-24 01:55:00,771 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e9f6045c839a421db18a8ccaf7ac24e4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4 2023-03-24 01:55:00,775 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4, entries=150, sequenceid=252, filesize=11.8 K 2023-03-24 01:55:00,775 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/bd43257641204d44b0c9fa8d8c6388ff as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff 2023-03-24 01:55:00,781 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff, entries=150, sequenceid=252, filesize=11.8 K 2023-03-24 01:55:00,781 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for d337c30c8fcf8ea343af6ac65f0aa65a in 1248ms, sequenceid=252, compaction requested=true 2023-03-24 01:55:00,781 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:00,781 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:00,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:00,784 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648512f2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:00,786 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:00,786 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,789 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:00,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,789 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:00,790 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:00,793 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:00,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:00,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:00,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:00,793 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:00,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:00,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:00,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,797 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,802 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:00,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:00,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:00,802 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:00,802 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:00,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:00,802 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:00,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:00,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:00,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:00,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:00,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:00,902 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:00,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:00,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:00,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:00,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:00,906 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:00,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:00,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:00,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:00,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:00,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,908 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:00,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:00,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:00,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:00,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:00,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:00,910 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:00,911 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:00,911 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:00,911 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:00,911 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:01,318 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/47e4b0f11c4d4dd691e8faff9cbd734b 2023-03-24 01:55:01,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:01,723 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:01,728 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/ba5147aa88b24238bdc9e5e5ebda6e73 2023-03-24 01:55:01,747 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/2979c3f5775c453a801292653ec9c07f 2023-03-24 01:55:01,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622961740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622961744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,753 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/47e4b0f11c4d4dd691e8faff9cbd734b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b 2023-03-24 01:55:01,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622961748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622961748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622961748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,757 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b, entries=150, sequenceid=267, filesize=11.9 K 2023-03-24 01:55:01,758 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/ba5147aa88b24238bdc9e5e5ebda6e73 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73 2023-03-24 01:55:01,769 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73, entries=150, sequenceid=267, filesize=11.9 K 2023-03-24 01:55:01,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/2979c3f5775c453a801292653ec9c07f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f 2023-03-24 01:55:01,782 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f, entries=150, sequenceid=267, filesize=11.9 K 2023-03-24 01:55:01,783 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for d337c30c8fcf8ea343af6ac65f0aa65a in 873ms, sequenceid=267, compaction requested=true 2023-03-24 01:55:01,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:01,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:01,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:01,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:01,784 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:01,785 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:01,786 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:01,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523b3fc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:01,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:01,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,795 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:01,795 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:01,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:01,801 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:01,801 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:01,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:01,805 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:01,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:01,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:01,805 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:01,805 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:01,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:01,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:01,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:01,813 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:01,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:01,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:01,813 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:01,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:01,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:01,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:01,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:55:01,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:01,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:01,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:01,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:01,853 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:01,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:01,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622961866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622961867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622961868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622961869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622961872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/b44c15bffeef47e68f29a926f3ff9a11 2023-03-24 01:55:01,914 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:01,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:01,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:01,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:01,917 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:01,917 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:01,917 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:01,917 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:01,917 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:01,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:01,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:01,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:01,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:01,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:01,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:01,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:01,921 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:01,921 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:01,924 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:01,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44408e3a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:01,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:01,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,927 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:01,927 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,927 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:01,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:01,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:01,931 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:01,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:01,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:01,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:01,931 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:01,932 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,932 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:01,932 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:01,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,933 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,937 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,937 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:01,937 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:01,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:01,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:01,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,938 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:01,938 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:01,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:01,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:01,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622961973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622961974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622961974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622961975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:01,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622961977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,038 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,043 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,045 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,047 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,047 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,047 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,047 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,049 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,049 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d9afb1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,051 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,051 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,053 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,053 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,058 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,058 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,060 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,060 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,064 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:02,064 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,064 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,165 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,169 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,169 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,171 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,174 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,174 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,174 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,174 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,176 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,177 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622962177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d873dfa[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,179 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622962178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622962178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622962178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,184 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,184 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,184 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,184 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,184 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622962181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:02,197 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,197 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,297 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,300 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,300 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,301 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,301 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,301 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,301 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,302 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,304 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,304 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,304 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,304 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,304 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/2bef40bba2a5464bab722323e661a1b4 2023-03-24 01:55:02,306 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a3d9e5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,310 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,318 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,318 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,318 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,318 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,322 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:02,323 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,323 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,423 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,426 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,428 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,430 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,432 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,434 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,434 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552ae35[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,435 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,440 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,440 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,441 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,441 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,441 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,442 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,448 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,449 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,449 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,449 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,449 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622962481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622962483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622962483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622962483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:02,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622962486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,549 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,552 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,552 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,552 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,552 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,552 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,553 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,555 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,556 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592a2d09[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,561 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,566 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,566 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,569 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,579 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,579 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,579 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,580 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,580 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,680 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,684 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,685 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,685 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,688 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,689 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,689 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,689 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:02,689 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:02,691 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:02,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d26b33b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:02,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:02,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:02,694 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,695 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:02,696 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:02,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:02,699 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:02,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:02,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:02,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:02,699 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:02,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:02,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:02,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:02,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:02,709 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:02,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:02,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,709 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:02,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:02,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:02,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:02,716 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/a640604824fd4ca9891dad79cc27dc4b 2023-03-24 01:55:02,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/b44c15bffeef47e68f29a926f3ff9a11 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11 2023-03-24 01:55:02,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11, entries=150, sequenceid=286, filesize=11.9 K 2023-03-24 01:55:02,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/2bef40bba2a5464bab722323e661a1b4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4 2023-03-24 01:55:02,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4, entries=150, sequenceid=286, filesize=11.9 K 2023-03-24 01:55:02,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/a640604824fd4ca9891dad79cc27dc4b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b 2023-03-24 01:55:02,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b, entries=150, sequenceid=286, filesize=11.9 K 2023-03-24 01:55:02,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d337c30c8fcf8ea343af6ac65f0aa65a in 882ms, sequenceid=286, compaction requested=true 2023-03-24 01:55:02,734 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:02,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:02,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:02,734 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:55:02,734 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:55:02,734 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:55:02,736 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72824 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:55:02,736 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:02,736 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,736 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/293e24c2c5d34ee2bd4275b2a8526d6d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=71.1 K 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72824 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 293e24c2c5d34ee2bd4275b2a8526d6d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:55:02,737 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,737 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1ba1b199dc29400e9cc8768342c699c6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=71.1 K 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6234e2a489654728a5d00e2a19dcc29b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679622897026 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1ba1b199dc29400e9cc8768342c699c6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:55:02,737 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting aa94c84733fc413180744793b76b2c0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679622898155 2023-03-24 01:55:02,738 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 20f3a705c06844f1b572ad1703587428, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679622897026 2023-03-24 01:55:02,738 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e9f6045c839a421db18a8ccaf7ac24e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679622899281 2023-03-24 01:55:02,739 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting dbecaece114249f7a2f3bb6c3524e484, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679622898155 2023-03-24 01:55:02,739 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ba5147aa88b24238bdc9e5e5ebda6e73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679622899596 2023-03-24 01:55:02,739 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f340cb89eb82421c84fb8232f736e25b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679622899281 2023-03-24 01:55:02,739 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2bef40bba2a5464bab722323e661a1b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:02,739 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 47e4b0f11c4d4dd691e8faff9cbd734b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679622899596 2023-03-24 01:55:02,740 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b44c15bffeef47e68f29a926f3ff9a11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:02,751 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#235 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:02,751 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#234 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:02,809 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:02,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:02,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:02,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:02,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:02,813 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:02,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:02,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:02,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,814 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:02,814 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:02,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:02,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:02,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:02,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:02,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:02,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:02,816 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:02,817 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:02,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:02,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:02,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622963002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34880 deadline: 1679622963004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622963005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622963005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34920 deadline: 1679622963006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622963108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622963109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34906 deadline: 1679622963110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622963110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622963111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,170 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/8051834575e24acc81552d76f75808ec as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8051834575e24acc81552d76f75808ec 2023-03-24 01:55:03,175 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/8b16e449d37142e58c0747450149a8f8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b16e449d37142e58c0747450149a8f8 2023-03-24 01:55:03,176 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 8051834575e24acc81552d76f75808ec(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:03,176 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:03,176 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=10, startTime=1679622902734; duration=0sec 2023-03-24 01:55:03,176 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:03,176 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:55:03,178 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72824 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:55:03,178 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:03,178 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,178 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/234ee3adc208446cbaaef33dfab9748c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=71.1 K 2023-03-24 01:55:03,179 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 234ee3adc208446cbaaef33dfab9748c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679622896398 2023-03-24 01:55:03,179 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c5854221a55248b791623bdaeb3bcf1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679622897026 2023-03-24 01:55:03,179 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5ba0866736924f62b5f54faa1bbc8e7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679622898155 2023-03-24 01:55:03,180 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting bd43257641204d44b0c9fa8d8c6388ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1679622899281 2023-03-24 01:55:03,180 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 8b16e449d37142e58c0747450149a8f8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:03,180 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:03,180 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=10, startTime=1679622902734; duration=0sec 2023-03-24 01:55:03,180 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2979c3f5775c453a801292653ec9c07f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679622899596 2023-03-24 01:55:03,180 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:03,181 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a640604824fd4ca9891dad79cc27dc4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:03,190 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#237 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:03,225 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/794b144d3ad94c518090bd51fa2b21f4 2023-03-24 01:55:03,253 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/f2d11b00680343b4b393360157ec65db 2023-03-24 01:55:03,263 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/69df16e8c66d4d7a84c5d06f9f17bf36 2023-03-24 01:55:03,267 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/794b144d3ad94c518090bd51fa2b21f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4 2023-03-24 01:55:03,270 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4, entries=150, sequenceid=299, filesize=11.9 K 2023-03-24 01:55:03,271 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/f2d11b00680343b4b393360157ec65db as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db 2023-03-24 01:55:03,276 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db, entries=150, sequenceid=299, filesize=11.9 K 2023-03-24 01:55:03,277 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/69df16e8c66d4d7a84c5d06f9f17bf36 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36 2023-03-24 01:55:03,284 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36, entries=150, sequenceid=299, filesize=11.9 K 2023-03-24 01:55:03,285 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d337c30c8fcf8ea343af6ac65f0aa65a in 469ms, sequenceid=299, compaction requested=false 2023-03-24 01:55:03,285 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:03,285 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,285 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6691398b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,291 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,292 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,293 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,296 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,296 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,298 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,298 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,298 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,300 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,302 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,302 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,302 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,302 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,303 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,303 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,303 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:03,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:03,318 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:03,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34910 deadline: 1679622963324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34912 deadline: 1679622963325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622963325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622963326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622963326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,403 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:03,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:03,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:03,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:03,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:03,406 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:03,406 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,406 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:03,406 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:03,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,407 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,409 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,409 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:03,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,413 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7622106a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,415 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,416 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,416 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,419 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,419 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,419 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:03,425 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,426 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,426 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622963427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622963430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622963430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622963430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622963430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,526 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:03,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:03,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:03,529 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:03,529 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,529 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:03,529 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:03,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,533 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,533 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:03,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:03,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:03,535 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,535 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:03,535 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,535 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,535 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,539 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,539 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,540 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee829c2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,543 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,543 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,547 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,547 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,547 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,547 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,547 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,548 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,549 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,553 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:03,554 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,554 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,554 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,599 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/5baa89539eef4a8ea2dfab656f2d939d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5baa89539eef4a8ea2dfab656f2d939d 2023-03-24 01:55:03,602 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 5baa89539eef4a8ea2dfab656f2d939d(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:03,602 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:03,603 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=10, startTime=1679622902734; duration=0sec 2023-03-24 01:55:03,603 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:03,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622963631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622963633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622963633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622963635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622963635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,654 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:03,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:03,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:03,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:03,659 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:03,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:03,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:03,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,661 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:03,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:03,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:03,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:03,662 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,662 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:03,663 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,663 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76dff24f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,671 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,675 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,675 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,680 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,687 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,687 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,687 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,687 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d085615da5b340f6808636dacfc01a1e 2023-03-24 01:55:03,787 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:03,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:03,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:03,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:03,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:03,790 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:03,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:03,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:03,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,791 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,792 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:03,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:03,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,793 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:03,794 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,794 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:03,794 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,794 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,794 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,800 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196dd07e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,802 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,806 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,806 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,807 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,807 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,808 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,812 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,813 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,813 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,913 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:03,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:03,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:03,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:03,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:03,916 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:03,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:03,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:03,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:03,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:03,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:03,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:03,919 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:03,920 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,920 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:03,920 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:03,920 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:03,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:03,924 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,924 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:03,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f2a3279[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:03,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:03,926 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,928 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:03,928 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,928 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:03,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:03,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:03,932 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:03,932 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:03,932 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:03,932 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:03,932 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:03,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:03,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,934 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:03,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:03,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622963935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622963935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622963935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:03,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:03,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:03,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:03,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:03,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:03,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,939 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:03,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:03,939 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:03,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:03,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:03,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622963940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:03,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622963940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,039 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:04,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:04,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:04,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:04,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:04,045 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:04,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:04,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:04,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,048 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:04,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:04,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:04,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:04,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,050 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:04,051 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:04,051 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:04,051 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:04,052 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,052 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:04,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b3b267[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:04,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:04,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:04,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:04,057 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:04,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,057 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:04,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:04,059 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:04,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:04,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:04,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:04,059 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:04,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:04,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:04,062 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:04,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:04,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:04,066 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:04,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:04,066 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:04,067 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:04,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/b5e689dbf19144258a508b84ed0bc0ab 2023-03-24 01:55:04,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/e40610656456498ba3cc5a0c751198b3 2023-03-24 01:55:04,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d085615da5b340f6808636dacfc01a1e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e 2023-03-24 01:55:04,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e, entries=200, sequenceid=322, filesize=14.3 K 2023-03-24 01:55:04,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/b5e689dbf19144258a508b84ed0bc0ab as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab 2023-03-24 01:55:04,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab, entries=150, sequenceid=322, filesize=11.9 K 2023-03-24 01:55:04,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/e40610656456498ba3cc5a0c751198b3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3 2023-03-24 01:55:04,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3, entries=150, sequenceid=322, filesize=11.9 K 2023-03-24 01:55:04,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for d337c30c8fcf8ea343af6ac65f0aa65a in 848ms, sequenceid=322, compaction requested=true 2023-03-24 01:55:04,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:04,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:04,164 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:04,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:04,164 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:04,164 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:04,165 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:04,165 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:04,165 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,165 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8051834575e24acc81552d76f75808ec, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=38.6 K 2023-03-24 01:55:04,166 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:04,166 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8051834575e24acc81552d76f75808ec, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:04,166 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:04,166 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,166 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b16e449d37142e58c0747450149a8f8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.2 K 2023-03-24 01:55:04,166 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 794b144d3ad94c518090bd51fa2b21f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622901867 2023-03-24 01:55:04,166 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8b16e449d37142e58c0747450149a8f8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:04,167 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d085615da5b340f6808636dacfc01a1e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:04,167 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f2d11b00680343b4b393360157ec65db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622901867 2023-03-24 01:55:04,167 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:04,167 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b5e689dbf19144258a508b84ed0bc0ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:04,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:04,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:04,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:04,170 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:04,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:04,170 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:04,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,171 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,172 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:04,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:04,173 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:04,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:04,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:04,174 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:04,174 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:04,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:04,175 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:55:04,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:04,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:04,176 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:04,187 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/2768075d11874ecebfe0e86610f92f7e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2768075d11874ecebfe0e86610f92f7e 2023-03-24 01:55:04,194 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 2768075d11874ecebfe0e86610f92f7e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:04,194 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:04,195 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622904164; duration=0sec 2023-03-24 01:55:04,195 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:04,195 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:04,195 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/fb8920baf17b4c3c8163c325584c43e7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/fb8920baf17b4c3c8163c325584c43e7 2023-03-24 01:55:04,196 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:04,196 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:04,196 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,196 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5baa89539eef4a8ea2dfab656f2d939d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.2 K 2023-03-24 01:55:04,196 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5baa89539eef4a8ea2dfab656f2d939d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679622901739 2023-03-24 01:55:04,197 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 69df16e8c66d4d7a84c5d06f9f17bf36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622901867 2023-03-24 01:55:04,197 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e40610656456498ba3cc5a0c751198b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:04,201 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into fb8920baf17b4c3c8163c325584c43e7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:04,201 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:04,201 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622904164; duration=0sec 2023-03-24 01:55:04,201 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:04,201 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb 2023-03-24 01:55:04,204 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:04,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:04,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622904475, max timestamp=1679622904477Num uniques 859; 2023-03-24 01:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622904475, max timestamp=1679622904477Num uniques 859; 2023-03-24 01:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622904475, max timestamp=1679622904477Num uniques 859; 2023-03-24 01:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:04,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622964487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622964490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622964491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622964491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622964492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622964592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622964595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622964595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622964595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622964596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,618 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/459a4630acbb4eb89d16ebecc4f9d275 2023-03-24 01:55:04,619 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/9b34cd738f92478897fb875c70f1f978 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b34cd738f92478897fb875c70f1f978 2023-03-24 01:55:04,624 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 9b34cd738f92478897fb875c70f1f978(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:04,624 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:04,624 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622904164; duration=0sec 2023-03-24 01:55:04,624 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:04,631 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6d929fabb05b4aaabfdf359af6dc6476 2023-03-24 01:55:04,635 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb 2023-03-24 01:55:04,638 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb, entries=150, sequenceid=335, filesize=11.9 K 2023-03-24 01:55:04,639 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/459a4630acbb4eb89d16ebecc4f9d275 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275 2023-03-24 01:55:04,643 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275, entries=150, sequenceid=335, filesize=11.9 K 2023-03-24 01:55:04,643 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6d929fabb05b4aaabfdf359af6dc6476 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476 2023-03-24 01:55:04,648 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476, entries=150, sequenceid=335, filesize=11.9 K 2023-03-24 01:55:04,649 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for d337c30c8fcf8ea343af6ac65f0aa65a in 474ms, sequenceid=335, compaction requested=false 2023-03-24 01:55:04,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:04,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:04,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:04,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:04,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:04,651 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:04,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f76db4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:04,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:04,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,659 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:04,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,660 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:04,661 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:04,664 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:04,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:04,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:04,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:04,664 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:04,666 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,666 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:04,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,666 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:04,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:04,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,669 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:04,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:04,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:04,671 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,671 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,672 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,672 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:04,672 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:04,672 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,672 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:04,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:04,772 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:04,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:04,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:04,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:04,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:04,775 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:04,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:04,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:04,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:04,781 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:04,781 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:04,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:04,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:04,783 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:04,783 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:04,796 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/392124cb5a874d50b49ad18fac15c07b 2023-03-24 01:55:04,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:04,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:04,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622964803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622964803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622964807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622964808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622964808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622964909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622964909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622964910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622964913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:04,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:04,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622964913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622965112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622965114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622965114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622965116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622965117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,210 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/f6f7188eabe94c37b4f6cb25065c22e8 2023-03-24 01:55:05,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622965415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622965418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622965418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622965419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622965420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,620 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6900c8fbc18f4ec4a59d07740a0a2fc6 2023-03-24 01:55:05,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/392124cb5a874d50b49ad18fac15c07b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b 2023-03-24 01:55:05,628 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b, entries=150, sequenceid=373, filesize=11.9 K 2023-03-24 01:55:05,629 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/f6f7188eabe94c37b4f6cb25065c22e8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8 2023-03-24 01:55:05,632 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8, entries=150, sequenceid=373, filesize=11.9 K 2023-03-24 01:55:05,633 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/6900c8fbc18f4ec4a59d07740a0a2fc6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6 2023-03-24 01:55:05,636 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6, entries=150, sequenceid=373, filesize=11.9 K 2023-03-24 01:55:05,637 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for d337c30c8fcf8ea343af6ac65f0aa65a in 854ms, sequenceid=373, compaction requested=true 2023-03-24 01:55:05,637 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:05,637 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:05,637 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:05,637 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:05,637 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,637 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:05,637 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:05,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,639 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,640 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:05,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6ebad4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:05,641 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:05,641 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,642 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:05,642 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:05,642 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:05,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:05,651 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:05,651 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:05,651 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:05,651 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:05,651 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:05,653 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,653 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,653 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,654 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:05,657 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:05,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,657 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,658 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:05,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,658 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:05,758 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:05,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:05,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:05,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:05,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:05,761 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:05,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:05,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:05,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:05,762 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,762 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,763 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:05,763 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:05,764 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,764 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:05,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:05,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:05,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:05,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:05,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:05,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:05,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:05,766 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:05,766 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:05,766 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,775 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/8f10ea0cfd03419fb0c26621dfde670d 2023-03-24 01:55:05,791 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/4e9a8ac45b4448fdbb0050d9c239c2c4 2023-03-24 01:55:05,807 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/8e193899ce234c7eaff47c6ef72ad87c 2023-03-24 01:55:05,811 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/8f10ea0cfd03419fb0c26621dfde670d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d 2023-03-24 01:55:05,814 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d, entries=150, sequenceid=385, filesize=11.9 K 2023-03-24 01:55:05,815 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/4e9a8ac45b4448fdbb0050d9c239c2c4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4 2023-03-24 01:55:05,818 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4, entries=150, sequenceid=385, filesize=11.9 K 2023-03-24 01:55:05,818 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/8e193899ce234c7eaff47c6ef72ad87c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c 2023-03-24 01:55:05,821 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c, entries=150, sequenceid=385, filesize=11.9 K 2023-03-24 01:55:05,822 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for d337c30c8fcf8ea343af6ac65f0aa65a in 56ms, sequenceid=385, compaction requested=true 2023-03-24 01:55:05,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:05,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,825 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,825 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1079ee9a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:05,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:05,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,828 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:05,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:05,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:05,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:05,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:05,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:05,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:05,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:05,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:05,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:05,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:05,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:05,842 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:05,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:05,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,843 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:05,843 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:05,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:05,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:05,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:05,943 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:05,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:05,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:05,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:05,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:05,946 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:05,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:05,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:05,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:05,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,947 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:05,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:05,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:05,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:05,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:05,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:05,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:05,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:05,952 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:05,952 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:05,952 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:05,952 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:05,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:05,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:05,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:05,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d044678[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:05,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,956 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:05,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,956 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:05,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:05,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:05,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:05,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:05,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:05,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:05,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:05,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:05,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:05,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:05,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:05,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:05,966 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622905934, max timestamp=1679622905967Num uniques 859; 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622905965, max timestamp=1679622905967Num uniques 859; 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622905965, max timestamp=1679622905967Num uniques 859; 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:05,969 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:05,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:05,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:05,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:05,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:05,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:05,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:05,972 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:05,972 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:05,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:05,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622965981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622965983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622965984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622965985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:05,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:05,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622965986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,073 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,076 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,081 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,083 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fec18a1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,087 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,087 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622966086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622966089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622966089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622966089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622966091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,094 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:06,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,097 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,097 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,198 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,201 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749fbd92[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:06,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,219 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,219 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,219 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,221 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,224 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,224 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622966289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622966293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622966293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622966293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622966295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,324 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,327 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b4d90a5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,338 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:06,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/31f38ce434364985ab96a20c4c709cfc 2023-03-24 01:55:06,340 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,340 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,350 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,350 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,450 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,453 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,461 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44152432[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,465 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,465 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:06,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,469 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,470 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,480 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,480 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,580 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,583 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,585 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,586 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,586 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b631501[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:06,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,595 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,595 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622966594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622966598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622966598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622966598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:06,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622966599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,602 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:06,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,602 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,602 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,603 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,703 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,706 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,707 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,710 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,710 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,710 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:06,711 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:06,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:06,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121948d2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:06,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,716 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:06,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:06,718 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:06,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:06,720 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:06,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:06,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:06,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:06,720 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:06,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:06,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,723 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,726 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:06,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,727 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:06,727 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:06,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:06,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:06,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/a3f518504f7a49cab30654bd1b5a191f 2023-03-24 01:55:06,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/10e9139898464ab0aa247b938f0ed408 2023-03-24 01:55:06,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/31f38ce434364985ab96a20c4c709cfc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc 2023-03-24 01:55:06,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc, entries=100, sequenceid=395, filesize=9.5 K 2023-03-24 01:55:06,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/a3f518504f7a49cab30654bd1b5a191f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f 2023-03-24 01:55:06,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f, entries=100, sequenceid=395, filesize=9.5 K 2023-03-24 01:55:06,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/10e9139898464ab0aa247b938f0ed408 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408 2023-03-24 01:55:06,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408, entries=100, sequenceid=395, filesize=9.5 K 2023-03-24 01:55:06,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for d337c30c8fcf8ea343af6ac65f0aa65a in 855ms, sequenceid=395, compaction requested=true 2023-03-24 01:55:06,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:06,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:06,781 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:55:06,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:06,781 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:55:06,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:06,782 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:06,783 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:55:06,783 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/fb8920baf17b4c3c8163c325584c43e7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=57.7 K 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:06,783 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,783 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2768075d11874ecebfe0e86610f92f7e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=57.7 K 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting fb8920baf17b4c3c8163c325584c43e7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2768075d11874ecebfe0e86610f92f7e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting cfb6c6eb7e6c4f24864e796aaa0d8bbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679622903324 2023-03-24 01:55:06,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 459a4630acbb4eb89d16ebecc4f9d275, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679622903324 2023-03-24 01:55:06,784 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 392124cb5a874d50b49ad18fac15c07b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622904490 2023-03-24 01:55:06,784 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f6f7188eabe94c37b4f6cb25065c22e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622904490 2023-03-24 01:55:06,784 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8f10ea0cfd03419fb0c26621dfde670d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679622904801 2023-03-24 01:55:06,784 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4e9a8ac45b4448fdbb0050d9c239c2c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679622904801 2023-03-24 01:55:06,785 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 31f38ce434364985ab96a20c4c709cfc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622905923 2023-03-24 01:55:06,785 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a3f518504f7a49cab30654bd1b5a191f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622905923 2023-03-24 01:55:06,793 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#258 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:06,794 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:06,806 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/26490e5464454b36bc6046b507371ed6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/26490e5464454b36bc6046b507371ed6 2023-03-24 01:55:06,811 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 26490e5464454b36bc6046b507371ed6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:06,811 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:06,811 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=11, startTime=1679622906781; duration=0sec 2023-03-24 01:55:06,811 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:06,811 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:55:06,813 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:55:06,819 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:06,819 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,819 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b34cd738f92478897fb875c70f1f978, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=57.7 K 2023-03-24 01:55:06,819 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9b34cd738f92478897fb875c70f1f978, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622903001 2023-03-24 01:55:06,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6d929fabb05b4aaabfdf359af6dc6476, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1679622903324 2023-03-24 01:55:06,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6900c8fbc18f4ec4a59d07740a0a2fc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622904490 2023-03-24 01:55:06,820 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/31572fc002094f46ac28add3e3dda30a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31572fc002094f46ac28add3e3dda30a 2023-03-24 01:55:06,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8e193899ce234c7eaff47c6ef72ad87c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1679622904801 2023-03-24 01:55:06,821 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 10e9139898464ab0aa247b938f0ed408, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622905923 2023-03-24 01:55:06,827 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:06,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:06,829 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:06,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:06,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,833 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,833 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 31572fc002094f46ac28add3e3dda30a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:06,833 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:06,833 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=11, startTime=1679622906781; duration=0sec 2023-03-24 01:55:06,833 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:06,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:06,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:06,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:06,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:06,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:06,840 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/4e7c11d35ae04159bb5bb29a11f244f1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4e7c11d35ae04159bb5bb29a11f244f1 2023-03-24 01:55:06,845 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 4e7c11d35ae04159bb5bb29a11f244f1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:06,845 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:06,845 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=11, startTime=1679622906781; duration=0sec 2023-03-24 01:55:06,846 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:06,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:06,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:06,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:06,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:06,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:06,852 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-03-24 01:55:06,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:06,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:06,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:06,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:06,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:06,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:06,866 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/aac7c844a8a64a92b65f98ffc2689d40 2023-03-24 01:55:07,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:07,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:07,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622967106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622967108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622967109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622967109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622967110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622967211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622967212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622967212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622967214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622967214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,277 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/1efd1b755f3f4b04bc44631b9c7843d1 2023-03-24 01:55:07,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622967415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622967416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622967417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622967418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622967418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,688 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/35bdf9f920d04cc4aa4cddb98cb5f1f3 2023-03-24 01:55:07,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/aac7c844a8a64a92b65f98ffc2689d40 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40 2023-03-24 01:55:07,696 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40, entries=150, sequenceid=434, filesize=11.9 K 2023-03-24 01:55:07,697 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/1efd1b755f3f4b04bc44631b9c7843d1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1 2023-03-24 01:55:07,700 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1, entries=150, sequenceid=434, filesize=11.9 K 2023-03-24 01:55:07,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/35bdf9f920d04cc4aa4cddb98cb5f1f3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3 2023-03-24 01:55:07,704 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3, entries=150, sequenceid=434, filesize=11.9 K 2023-03-24 01:55:07,705 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=53.67 KB/54960 for d337c30c8fcf8ea343af6ac65f0aa65a in 853ms, sequenceid=434, compaction requested=false 2023-03-24 01:55:07,705 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:07,705 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:07,705 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:07,705 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:07,705 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,705 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:07,705 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:07,707 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,707 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:07,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55963a8c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:07,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:07,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:07,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:07,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:07,712 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,712 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:07,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:07,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:07,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:07,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:07,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:07,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:07,716 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,716 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,716 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,718 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:07,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:07,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:07,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:55:07,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:07,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:07,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,721 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:07,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:07,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:07,722 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:07,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:07,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:07,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:07,722 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:07,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622967755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622967756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622967759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622967760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622967760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,822 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:07,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:07,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:07,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:07,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:07,825 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:07,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:07,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:07,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:07,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,830 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:07,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:07,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:07,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:07,832 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:07,832 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:07,832 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:07,832 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:07,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:07,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384fb0b8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:07,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:07,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,837 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:07,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:07,838 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:07,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,839 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:07,839 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,839 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:07,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:07,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:07,840 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:07,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:07,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:07,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:07,840 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:07,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,842 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:07,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:07,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:07,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:07,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,849 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:07,849 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:07,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622967861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622967861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622967863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622967863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:07,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622967863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,950 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:07,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:07,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:07,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:07,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:07,953 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:07,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:07,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:07,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:07,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:07,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:07,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:07,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:07,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:07,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:07,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:07,957 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:07,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:07,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:07,963 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:07,964 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:07,964 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:07,964 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:07,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:07,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:07,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:07,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:07,966 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,966 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:07,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b8771ad[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:07,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:07,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:07,971 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:07,974 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:07,974 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:07,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:07,980 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:07,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:07,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:07,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:07,981 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:07,982 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,982 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,982 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:07,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:07,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,984 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:07,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:07,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:07,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:07,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:07,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:07,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:07,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:07,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:07,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:07,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:07,987 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:07,987 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:08,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622968063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622968064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622968066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622968066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622968068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,087 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:08,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:08,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:08,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:08,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:08,090 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:08,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:08,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:08,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:08,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:08,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:08,093 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:08,094 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,094 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:08,094 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:08,094 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:08,094 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:08,096 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,096 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:08,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6300e1d7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:08,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:08,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:08,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,100 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:08,100 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:08,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:08,104 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:08,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:08,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:08,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:08,105 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:08,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,107 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:08,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,112 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,112 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:08,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,113 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,113 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/7a4e89559eb8421b93c71a91de306ff3 2023-03-24 01:55:08,213 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:08,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:08,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:08,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:08,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:08,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:08,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:08,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,217 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,217 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:08,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:08,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:08,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,219 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:08,220 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,220 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:08,220 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:08,220 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:08,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:08,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:08,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6f561[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:08,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:08,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:08,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:08,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:08,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:08,230 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:08,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:08,232 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:08,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:08,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:08,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:08,232 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:08,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:08,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:08,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,239 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:08,239 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:08,339 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:08,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:08,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:08,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:08,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:08,343 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:08,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:08,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:08,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:08,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:08,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:08,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:08,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:08,347 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:08,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:08,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:08,349 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:08,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a95f5d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:08,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:08,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:08,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:08,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:08,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:08,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:08,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:08,357 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:08,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:08,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:08,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:08,357 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:08,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,362 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:08,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:08,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,365 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:08,365 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:08,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622968367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622968368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622968370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622968370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622968371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,465 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:08,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:08,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:08,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:08,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:08,468 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:08,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:08,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:08,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,470 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:08,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:08,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:08,472 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,472 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:08,472 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:08,472 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:08,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:08,474 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:08,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@422344bc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:08,475 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:08,475 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:08,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:08,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:08,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:08,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:08,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:08,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:08,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:08,482 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:08,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:08,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:08,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:08,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:08,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:08,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:08,488 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:08,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,488 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:08,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:08,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/31903cd1d6214a7ba18d1dafc27dd52e 2023-03-24 01:55:08,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/407ccc1b71e24868a8068be685a0453f 2023-03-24 01:55:08,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/7a4e89559eb8421b93c71a91de306ff3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3 2023-03-24 01:55:08,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3, entries=100, sequenceid=447, filesize=9.5 K 2023-03-24 01:55:08,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/31903cd1d6214a7ba18d1dafc27dd52e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e 2023-03-24 01:55:08,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e, entries=100, sequenceid=447, filesize=9.5 K 2023-03-24 01:55:08,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/407ccc1b71e24868a8068be685a0453f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f 2023-03-24 01:55:08,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f, entries=100, sequenceid=447, filesize=9.5 K 2023-03-24 01:55:08,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for d337c30c8fcf8ea343af6ac65f0aa65a in 844ms, sequenceid=447, compaction requested=true 2023-03-24 01:55:08,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:08,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:08,566 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:08,566 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:08,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:08,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:08,567 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,567 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/26490e5464454b36bc6046b507371ed6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=34.1 K 2023-03-24 01:55:08,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31572fc002094f46ac28add3e3dda30a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=34.1 K 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 26490e5464454b36bc6046b507371ed6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622904807 2023-03-24 01:55:08,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 31572fc002094f46ac28add3e3dda30a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622904807 2023-03-24 01:55:08,568 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting aac7c844a8a64a92b65f98ffc2689d40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679622905983 2023-03-24 01:55:08,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1efd1b755f3f4b04bc44631b9c7843d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679622905983 2023-03-24 01:55:08,568 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7a4e89559eb8421b93c71a91de306ff3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622907109 2023-03-24 01:55:08,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 31903cd1d6214a7ba18d1dafc27dd52e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622907109 2023-03-24 01:55:08,575 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:08,577 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:08,588 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:08,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:08,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:08,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:08,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:08,594 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:08,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:08,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:08,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:08,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:08,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:08,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:08,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:08,601 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/2d1312a24d1149818dc14a87bd43b83e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/2d1312a24d1149818dc14a87bd43b83e 2023-03-24 01:55:08,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:08,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:08,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:08,606 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 2d1312a24d1149818dc14a87bd43b83e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:08,606 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:08,606 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622908566; duration=0sec 2023-03-24 01:55:08,607 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:08,607 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:08,608 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:08,608 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:08,608 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:08,608 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,608 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:08,608 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:08,609 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:08,609 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4e7c11d35ae04159bb5bb29a11f244f1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=34.1 K 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:08,610 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4e7c11d35ae04159bb5bb29a11f244f1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1679622904807 2023-03-24 01:55:08,610 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 35bdf9f920d04cc4aa4cddb98cb5f1f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1679622905983 2023-03-24 01:55:08,611 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 407ccc1b71e24868a8068be685a0453f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622907109 2023-03-24 01:55:08,618 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:08,652 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/89c05ed7144b467aae3d31d5c0dd0e76 2023-03-24 01:55:08,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:08,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:08,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622968878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622968879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622968880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622968880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622968882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622968984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622968985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622968985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622968985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:08,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622968985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:08,995 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/70b59cd96d544590a7689ca191c7915b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/70b59cd96d544590a7689ca191c7915b 2023-03-24 01:55:08,999 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 70b59cd96d544590a7689ca191c7915b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:08,999 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:08,999 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622908566; duration=0sec 2023-03-24 01:55:08,999 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:09,066 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/1e60b4dd1bfc45f29d947052c8bb3b1b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1e60b4dd1bfc45f29d947052c8bb3b1b 2023-03-24 01:55:09,070 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 1e60b4dd1bfc45f29d947052c8bb3b1b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:09,070 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:09,070 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622908566; duration=0sec 2023-03-24 01:55:09,070 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/96d7ebce865b448e96bdd79ec4d35989 2023-03-24 01:55:09,070 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:09,079 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/915a8e99784a4b8e9fafb18c0fcbbd32 2023-03-24 01:55:09,083 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/89c05ed7144b467aae3d31d5c0dd0e76 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76 2023-03-24 01:55:09,086 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76, entries=150, sequenceid=468, filesize=11.9 K 2023-03-24 01:55:09,087 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/96d7ebce865b448e96bdd79ec4d35989 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989 2023-03-24 01:55:09,093 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989, entries=150, sequenceid=468, filesize=11.9 K 2023-03-24 01:55:09,093 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/915a8e99784a4b8e9fafb18c0fcbbd32 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32 2023-03-24 01:55:09,096 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32, entries=150, sequenceid=468, filesize=11.9 K 2023-03-24 01:55:09,097 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d337c30c8fcf8ea343af6ac65f0aa65a in 488ms, sequenceid=468, compaction requested=false 2023-03-24 01:55:09,097 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:09,097 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,097 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26490b76[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,102 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,110 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,110 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:09,116 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,116 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:09,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:55:09,190 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:09,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:09,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:09,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:09,191 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:09,191 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:09,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/21456253dbef4b70bdeb1f0f19e2b9ca 2023-03-24 01:55:09,216 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,219 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,223 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,225 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,225 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f4ef23[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622969228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622969228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622969230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622969231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,234 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,234 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,234 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622969232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,237 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,237 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,239 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,239 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,239 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,241 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:09,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,244 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,244 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622969332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622969332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622969334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622969334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622969335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,344 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,347 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaf0917[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,363 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,367 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,368 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,375 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,375 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:09,375 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,475 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,478 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,482 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,490 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,490 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,490 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,490 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4671a8da[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,492 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,492 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,493 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,494 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,494 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,494 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,495 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,495 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,495 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,495 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,498 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,498 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,502 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,502 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,505 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,508 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,508 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622969535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622969535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622969535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622969536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622969537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,608 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/aafb0ecce7cf402ea880ad700b02c62b 2023-03-24 01:55:09,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,612 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,614 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,614 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,615 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,615 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,619 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,619 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,619 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,619 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,619 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288d19d9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,626 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,626 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,626 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,626 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,627 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,630 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,630 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,634 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,635 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,639 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,639 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,639 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,740 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,743 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,746 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,747 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,747 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,747 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,749 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,749 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,750 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,750 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b1661c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,751 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,751 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,752 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,755 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,755 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,755 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,757 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,757 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,757 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,757 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,757 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,763 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,768 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,768 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,778 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,781 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,781 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,782 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,782 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,783 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622969838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622969839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622969839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622969840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:09,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622969842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,882 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:09,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:09,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:09,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:09,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:09,885 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:09,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:09,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:09,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:09,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:09,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:09,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:09,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:09,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:09,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:09,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:09,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:09,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:09,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f30238c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:09,902 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:09,902 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:09,904 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,904 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:09,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:09,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:09,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:09,908 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:09,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:09,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:09,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:09,908 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:09,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,909 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:09,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:09,911 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,912 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:09,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:09,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:09,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:09,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:09,919 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:09,919 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:10,019 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:10,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:10,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:10,022 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:10,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:10,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:10,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:10,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/9b2ed388379a43339d1e6b718d005e61 2023-03-24 01:55:10,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:10,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:10,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:10,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:10,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:10,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:10,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:10,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:10,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:10,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:10,026 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:10,026 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,026 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:10,026 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:10,027 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:10,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/21456253dbef4b70bdeb1f0f19e2b9ca as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca 2023-03-24 01:55:10,028 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:10,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:10,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:10,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@344c73c8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:10,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:10,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca, entries=150, sequenceid=484, filesize=11.9 K 2023-03-24 01:55:10,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/aafb0ecce7cf402ea880ad700b02c62b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b 2023-03-24 01:55:10,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,034 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:10,035 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,035 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:10,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:10,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b, entries=150, sequenceid=484, filesize=11.9 K 2023-03-24 01:55:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/9b2ed388379a43339d1e6b718d005e61 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61 2023-03-24 01:55:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:10,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:10,042 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:10,042 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:10,042 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:10,042 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:10,042 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:10,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61, entries=150, sequenceid=484, filesize=11.9 K 2023-03-24 01:55:10,044 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,044 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:10,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d337c30c8fcf8ea343af6ac65f0aa65a in 854ms, sequenceid=484, compaction requested=true 2023-03-24 01:55:10,044 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:10,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:10,045 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:10,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:10,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:10,045 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:10,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:10,045 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,047 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:10,047 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:10,047 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:10,047 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:10,047 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,047 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/70b59cd96d544590a7689ca191c7915b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.6 K 2023-03-24 01:55:10,047 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,048 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/2d1312a24d1149818dc14a87bd43b83e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.6 K 2023-03-24 01:55:10,048 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2d1312a24d1149818dc14a87bd43b83e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622905984 2023-03-24 01:55:10,048 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 70b59cd96d544590a7689ca191c7915b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622905984 2023-03-24 01:55:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,049 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 89c05ed7144b467aae3d31d5c0dd0e76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679622907727 2023-03-24 01:55:10,049 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 96d7ebce865b448e96bdd79ec4d35989, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679622907727 2023-03-24 01:55:10,049 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,049 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 21456253dbef4b70bdeb1f0f19e2b9ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:10,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:10,050 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting aafb0ecce7cf402ea880ad700b02c62b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:10,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:10,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,054 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:10,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:10,054 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:10,054 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:10,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:10,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:10,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:10,060 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:10,061 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:10,074 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/0c2ad3d4bedd456fa3929180cd9c5171 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/0c2ad3d4bedd456fa3929180cd9c5171 2023-03-24 01:55:10,079 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 0c2ad3d4bedd456fa3929180cd9c5171(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:10,079 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:10,079 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622910045; duration=0sec 2023-03-24 01:55:10,079 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:10,079 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:10,080 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:10,081 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:10,081 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,081 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1e60b4dd1bfc45f29d947052c8bb3b1b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.6 K 2023-03-24 01:55:10,081 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1e60b4dd1bfc45f29d947052c8bb3b1b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1679622905984 2023-03-24 01:55:10,082 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 915a8e99784a4b8e9fafb18c0fcbbd32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679622907727 2023-03-24 01:55:10,082 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9b2ed388379a43339d1e6b718d005e61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:10,096 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:10,154 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:10,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:10,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:10,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:10,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:10,158 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:10,158 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:10,158 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:10,158 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:10,159 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,159 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,161 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:10,161 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:10,162 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,162 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:10,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:10,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:10,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:10,163 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:10,163 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:10,163 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:10,163 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:10,163 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:10,163 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:10,163 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:10,164 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:10,164 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:10,181 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d5902687ac514cb5ae2413b5aef7a4c0 2023-03-24 01:55:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:10,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622970350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622970350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622970351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622970352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622970353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622970454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622970454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622970455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622970455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622970456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,475 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/5154477d4f694837952eb7c178568ff5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/5154477d4f694837952eb7c178568ff5 2023-03-24 01:55:10,479 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 5154477d4f694837952eb7c178568ff5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:10,479 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:10,479 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622910045; duration=0sec 2023-03-24 01:55:10,479 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:10,505 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/360ef6626ac34c149800b8996141711b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/360ef6626ac34c149800b8996141711b 2023-03-24 01:55:10,509 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 360ef6626ac34c149800b8996141711b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:10,509 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:10,509 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622910045; duration=0sec 2023-03-24 01:55:10,509 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:10,595 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3eea1ef01be4569a5dec713db5c7846 2023-03-24 01:55:10,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622970656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622970658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622970658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622970658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622970658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622970961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622970962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622970962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622970963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:10,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:10,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622970974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,005 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/1853ea726e884fa5b4a30b60b126fd7a 2023-03-24 01:55:11,010 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d5902687ac514cb5ae2413b5aef7a4c0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0 2023-03-24 01:55:11,013 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0, entries=150, sequenceid=504, filesize=11.9 K 2023-03-24 01:55:11,014 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/e3eea1ef01be4569a5dec713db5c7846 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846 2023-03-24 01:55:11,018 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846, entries=150, sequenceid=504, filesize=11.9 K 2023-03-24 01:55:11,018 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/1853ea726e884fa5b4a30b60b126fd7a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a 2023-03-24 01:55:11,022 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a, entries=150, sequenceid=504, filesize=11.9 K 2023-03-24 01:55:11,023 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d337c30c8fcf8ea343af6ac65f0aa65a in 859ms, sequenceid=504, compaction requested=false 2023-03-24 01:55:11,023 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:11,023 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:11,025 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,025 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:11,025 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,025 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:11,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:11,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744ff749[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:11,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:11,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,028 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,028 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:11,028 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:11,028 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:11,028 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,028 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:11,029 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:11,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:11,032 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:11,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:11,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:11,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:11,032 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:11,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:11,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:11,035 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:11,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:11,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:11,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:11,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:11,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:11,039 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:11,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:11,040 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:11,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:11,140 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:11,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:11,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:11,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:11,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:11,143 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:11,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:11,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:11,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:11,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:11,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:11,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:11,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:11,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:11,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:11,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:11,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:11,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:11,148 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:55:11,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:11,149 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:11,149 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:11,149 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:11,149 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:11,149 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:11,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:11,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:11,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622971478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622971478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622971479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622971480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622971481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,558 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/1be24c07e75946f999f5d23b8408e3df 2023-03-24 01:55:11,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622971582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622971583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622971583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622971584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622971786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622971786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622971786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:11,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622971787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:11,982 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/68e3f70983e54277b8ec20ad757ef717 2023-03-24 01:55:12,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622972088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622972088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622972090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622972091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,401 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/309cf79bc8d24a0baf204720158e30ed 2023-03-24 01:55:12,406 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/1be24c07e75946f999f5d23b8408e3df as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df 2023-03-24 01:55:12,412 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df, entries=150, sequenceid=519, filesize=11.9 K 2023-03-24 01:55:12,412 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/68e3f70983e54277b8ec20ad757ef717 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717 2023-03-24 01:55:12,416 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717, entries=150, sequenceid=519, filesize=11.9 K 2023-03-24 01:55:12,416 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/309cf79bc8d24a0baf204720158e30ed as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed 2023-03-24 01:55:12,420 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed, entries=150, sequenceid=519, filesize=11.9 K 2023-03-24 01:55:12,421 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d337c30c8fcf8ea343af6ac65f0aa65a in 1273ms, sequenceid=519, compaction requested=true 2023-03-24 01:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:12,421 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:12,421 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:12,423 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,423 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:12,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:12,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66dcee5a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:12,425 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:12,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,427 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,427 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:12,427 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:12,427 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:12,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:12,429 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:12,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:12,432 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:12,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:12,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:12,432 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:12,432 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:12,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:12,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:12,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:12,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:12,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,441 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:12,441 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:12,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:12,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:12,499 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:12,499 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:12,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:12,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d9ab890f5f7743c2bd9f6a03470147eb 2023-03-24 01:55:12,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/dc09a22d153a4571ad9fd918f979374a 2023-03-24 01:55:12,541 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:12,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:12,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:12,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622972541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:12,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:12,545 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:12,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:12,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:12,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:12,545 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,546 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,551 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:12,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:12,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:12,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:12,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:12,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:12,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:12,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:12,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:12,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/09d92ad520c64169bf71d24acd25a059 2023-03-24 01:55:12,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:12,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:12,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:12,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:12,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,575 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,576 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188e9d4e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:12,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:12,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/d9ab890f5f7743c2bd9f6a03470147eb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb 2023-03-24 01:55:12,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:12,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:12,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:12,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:12,580 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb, entries=200, sequenceid=539, filesize=14.3 K 2023-03-24 01:55:12,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/dc09a22d153a4571ad9fd918f979374a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a 2023-03-24 01:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:12,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:12,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a, entries=150, sequenceid=539, filesize=11.9 K 2023-03-24 01:55:12,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/09d92ad520c64169bf71d24acd25a059 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059 2023-03-24 01:55:12,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:12,586 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:12,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:12,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:12,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:12,588 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:12,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34912 deadline: 1679622972591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34906 deadline: 1679622972593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:12,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:12,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:12,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:12,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:12,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,596 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:12,596 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:12,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059, entries=150, sequenceid=539, filesize=11.9 K 2023-03-24 01:55:12,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d337c30c8fcf8ea343af6ac65f0aa65a in 99ms, sequenceid=539, compaction requested=true 2023-03-24 01:55:12,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:12,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:12,598 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:12,598 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:12,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:12,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:12,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:12,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:55:12,601 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52172 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:12,601 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:12,601 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,601 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/0c2ad3d4bedd456fa3929180cd9c5171, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=50.9 K 2023-03-24 01:55:12,601 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:12,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:12,601 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:12,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:12,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:12,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,601 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0c2ad3d4bedd456fa3929180cd9c5171, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:12,602 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,602 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/5154477d4f694837952eb7c178568ff5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=48.6 K 2023-03-24 01:55:12,603 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d5902687ac514cb5ae2413b5aef7a4c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679622909197 2023-03-24 01:55:12,604 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5154477d4f694837952eb7c178568ff5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:12,604 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1be24c07e75946f999f5d23b8408e3df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1679622910347 2023-03-24 01:55:12,607 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e3eea1ef01be4569a5dec713db5c7846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679622909197 2023-03-24 01:55:12,607 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d9ab890f5f7743c2bd9f6a03470147eb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:12,608 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 68e3f70983e54277b8ec20ad757ef717, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1679622910347 2023-03-24 01:55:12,609 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting dc09a22d153a4571ad9fd918f979374a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:12,633 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622972638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622972642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622972644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,649 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:12,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/77d6b18c8fd042c28345c8b3fa031319 2023-03-24 01:55:12,674 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/c9627ded7e034712a1d365fa0411159a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/c9627ded7e034712a1d365fa0411159a 2023-03-24 01:55:12,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/654335e783ef4a3595bdec2f570414de 2023-03-24 01:55:12,680 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into c9627ded7e034712a1d365fa0411159a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:12,680 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:12,680 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=12, startTime=1679622912598; duration=0sec 2023-03-24 01:55:12,680 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:12,680 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:12,682 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:12,682 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:12,682 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,682 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/360ef6626ac34c149800b8996141711b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=48.6 K 2023-03-24 01:55:12,683 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 360ef6626ac34c149800b8996141711b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1679622908881 2023-03-24 01:55:12,683 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1853ea726e884fa5b4a30b60b126fd7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679622909197 2023-03-24 01:55:12,683 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 309cf79bc8d24a0baf204720158e30ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1679622910347 2023-03-24 01:55:12,684 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 09d92ad520c64169bf71d24acd25a059, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:12,689 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:12,696 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:12,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:12,698 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/ce403f118046459a89acdf8d88da8260 2023-03-24 01:55:12,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:12,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:12,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:12,701 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,702 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,702 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:12,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:12,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/77d6b18c8fd042c28345c8b3fa031319 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319 2023-03-24 01:55:12,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:12,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319, entries=150, sequenceid=553, filesize=11.9 K 2023-03-24 01:55:12,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/654335e783ef4a3595bdec2f570414de as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de 2023-03-24 01:55:12,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:12,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. started... 2023-03-24 01:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. as already flushing 2023-03-24 01:55:12,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:12,708 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:12,709 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:12,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,710 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de, entries=150, sequenceid=553, filesize=11.9 K 2023-03-24 01:55:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/ce403f118046459a89acdf8d88da8260 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260 2023-03-24 01:55:12,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:12,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8b6fa4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:12,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:12,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:12,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:12,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:12,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:12,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260, entries=150, sequenceid=553, filesize=11.9 K 2023-03-24 01:55:12,717 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:12,717 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,717 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:12,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,717 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d337c30c8fcf8ea343af6ac65f0aa65a in 118ms, sequenceid=553, compaction requested=false 2023-03-24 01:55:12,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:12,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:12,719 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:12,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:12,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:12,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:12,719 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:12,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:12,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:12,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:12,725 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,726 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:12,726 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:12,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:12,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:12,726 DEBUG [Thread-1569] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:12,726 DEBUG [Thread-1569] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:12,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:12,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:55:12,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:12,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:12,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:12,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:12,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/86d3e7acfcad40f694a5f5e4898fe814 2023-03-24 01:55:12,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622972767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622972767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,788 DEBUG [Thread-1574] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:62686 2023-03-24 01:55:12,788 DEBUG [Thread-1574] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:12,788 DEBUG [Thread-1572] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:62686 2023-03-24 01:55:12,788 DEBUG [Thread-1572] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:12,791 DEBUG [Thread-1578] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:62686 2023-03-24 01:55:12,791 DEBUG [Thread-1570] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7810a231 to 127.0.0.1:62686 2023-03-24 01:55:12,791 DEBUG [Thread-1578] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:12,791 DEBUG [Thread-1570] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:12,792 DEBUG [Thread-1576] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:62686 2023-03-24 01:55:12,792 DEBUG [Thread-1576] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:12,826 DEBUG [Thread-1569] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:12,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:12,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622972847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622972873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:12,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:12,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622972873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,074 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/375c2fc18538462084df4278abdb871f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/375c2fc18538462084df4278abdb871f 2023-03-24 01:55:13,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:13,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622973074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:13,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622973075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,078 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into 375c2fc18538462084df4278abdb871f(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:13,078 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,078 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=12, startTime=1679622912598; duration=0sec 2023-03-24 01:55:13,078 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:13,099 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/794b6bbcfe5846c5b272ddf77c711fb8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/794b6bbcfe5846c5b272ddf77c711fb8 2023-03-24 01:55:13,103 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 794b6bbcfe5846c5b272ddf77c711fb8(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:13,103 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,103 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=12, startTime=1679622912599; duration=0sec 2023-03-24 01:55:13,103 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34920 deadline: 1679622973150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/ef4ff81b867642aba85c32f9caa21f61 2023-03-24 01:55:13,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:13,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34910 deadline: 1679622973377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:13,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34880 deadline: 1679622973377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/e20051c2125647eba6e2bd407c39e5c6 2023-03-24 01:55:13,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/86d3e7acfcad40f694a5f5e4898fe814 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814 2023-03-24 01:55:13,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814, entries=150, sequenceid=573, filesize=11.9 K 2023-03-24 01:55:13,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/ef4ff81b867642aba85c32f9caa21f61 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61 2023-03-24 01:55:13,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61, entries=150, sequenceid=573, filesize=11.9 K 2023-03-24 01:55:13,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/e20051c2125647eba6e2bd407c39e5c6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6 2023-03-24 01:55:13,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6, entries=150, sequenceid=573, filesize=11.9 K 2023-03-24 01:55:13,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for d337c30c8fcf8ea343af6ac65f0aa65a in 844ms, sequenceid=573, compaction requested=true 2023-03-24 01:55:13,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:13,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:13,588 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:13,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:13,588 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:13,589 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:13,589 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:13,589 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/A is initiating minor compaction (all files) 2023-03-24 01:55:13,589 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/B is initiating minor compaction (all files) 2023-03-24 01:55:13,589 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/A in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:13,589 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/B in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:13,589 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/375c2fc18538462084df4278abdb871f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.8 K 2023-03-24 01:55:13,589 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/c9627ded7e034712a1d365fa0411159a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.8 K 2023-03-24 01:55:13,590 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 375c2fc18538462084df4278abdb871f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:13,590 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c9627ded7e034712a1d365fa0411159a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:13,590 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 77d6b18c8fd042c28345c8b3fa031319, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1679622912534 2023-03-24 01:55:13,590 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 654335e783ef4a3595bdec2f570414de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1679622912534 2023-03-24 01:55:13,590 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 86d3e7acfcad40f694a5f5e4898fe814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679622912628 2023-03-24 01:55:13,591 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ef4ff81b867642aba85c32f9caa21f61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679622912628 2023-03-24 01:55:13,600 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#B#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:13,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:13,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:13,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:13,602 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#A#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:13,602 DEBUG [Thread-1559] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12e82edd to 127.0.0.1:62686 2023-03-24 01:55:13,603 DEBUG [Thread-1559] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,604 DEBUG [Thread-1565] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:62686 2023-03-24 01:55:13,604 DEBUG [Thread-1565] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,617 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/880529893eaa456cabf675881c68c814 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880529893eaa456cabf675881c68c814 2023-03-24 01:55:13,623 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/ebcd4f6d671741a2adc812efc23bad2c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/ebcd4f6d671741a2adc812efc23bad2c 2023-03-24 01:55:13,625 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/B of d337c30c8fcf8ea343af6ac65f0aa65a into 880529893eaa456cabf675881c68c814(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:13,625 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,625 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/B, priority=13, startTime=1679622913588; duration=0sec 2023-03-24 01:55:13,625 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:13,625 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:13,627 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:13,627 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): d337c30c8fcf8ea343af6ac65f0aa65a/C is initiating minor compaction (all files) 2023-03-24 01:55:13,627 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d337c30c8fcf8ea343af6ac65f0aa65a/C in TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:13,627 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/794b6bbcfe5846c5b272ddf77c711fb8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp, totalSize=36.8 K 2023-03-24 01:55:13,628 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 794b6bbcfe5846c5b272ddf77c711fb8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1679622911477 2023-03-24 01:55:13,628 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ce403f118046459a89acdf8d88da8260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1679622912534 2023-03-24 01:55:13,629 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/A of d337c30c8fcf8ea343af6ac65f0aa65a into ebcd4f6d671741a2adc812efc23bad2c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:13,629 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,629 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/A, priority=13, startTime=1679622913588; duration=0sec 2023-03-24 01:55:13,629 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e20051c2125647eba6e2bd407c39e5c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679622912628 2023-03-24 01:55:13,629 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:13,642 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): d337c30c8fcf8ea343af6ac65f0aa65a#C#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:13,654 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/38eed80928e44bd296e83dd1457ef389 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/38eed80928e44bd296e83dd1457ef389 2023-03-24 01:55:13,656 DEBUG [Thread-1563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:62686 2023-03-24 01:55:13,656 DEBUG [Thread-1563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,666 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d337c30c8fcf8ea343af6ac65f0aa65a/C of d337c30c8fcf8ea343af6ac65f0aa65a into 38eed80928e44bd296e83dd1457ef389(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:13,666 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:13,666 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a., storeName=d337c30c8fcf8ea343af6ac65f0aa65a/C, priority=13, startTime=1679622913588; duration=0sec 2023-03-24 01:55:13,666 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:13,879 DEBUG [Thread-1567] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:62686 2023-03-24 01:55:13,879 DEBUG [Thread-1567] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,882 DEBUG [Thread-1561] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:62686 2023-03-24 01:55:13,882 DEBUG [Thread-1561] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 5191 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4915 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 5187 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 5204 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4931 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:55:13,882 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:55:13,883 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7233cccd to 127.0.0.1:62686 2023-03-24 01:55:13,883 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:13,883 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:55:13,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:55:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:13,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:13,888 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622913888"}]},"ts":"1679622913888"} 2023-03-24 01:55:13,889 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:55:13,985 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:55:13,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, UNASSIGN}] 2023-03-24 01:55:13,988 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, UNASSIGN 2023-03-24 01:55:13,988 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d337c30c8fcf8ea343af6ac65f0aa65a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:13,988 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622913988"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622913988"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622913988"}]},"ts":"1679622913988"} 2023-03-24 01:55:13,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:13,989 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:14,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/72c2741d96944297bc243d1b3ed3122a 2023-03-24 01:55:14,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:14,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d337c30c8fcf8ea343af6ac65f0aa65a, disabling compactions & flushes 2023-03-24 01:55:14,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:14,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:14,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/880633672ff44f2e9c7f994040265129 2023-03-24 01:55:14,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:14,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/b64f0a32f576442fb8f6ec5f8ae3a634 2023-03-24 01:55:14,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/72c2741d96944297bc243d1b3ed3122a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/72c2741d96944297bc243d1b3ed3122a 2023-03-24 01:55:14,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/72c2741d96944297bc243d1b3ed3122a, entries=150, sequenceid=590, filesize=11.9 K 2023-03-24 01:55:14,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/880633672ff44f2e9c7f994040265129 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880633672ff44f2e9c7f994040265129 2023-03-24 01:55:14,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880633672ff44f2e9c7f994040265129, entries=150, sequenceid=590, filesize=11.9 K 2023-03-24 01:55:14,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/b64f0a32f576442fb8f6ec5f8ae3a634 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/b64f0a32f576442fb8f6ec5f8ae3a634 2023-03-24 01:55:14,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/b64f0a32f576442fb8f6ec5f8ae3a634, entries=150, sequenceid=590, filesize=11.9 K 2023-03-24 01:55:14,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=26.84 KB/27480 for d337c30c8fcf8ea343af6ac65f0aa65a in 1250ms, sequenceid=590, compaction requested=false 2023-03-24 01:55:14,852 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:14,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. after waiting 0 ms 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:14,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d337c30c8fcf8ea343af6ac65f0aa65a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=A 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=B 2023-03-24 01:55:14,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:14,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d337c30c8fcf8ea343af6ac65f0aa65a, store=C 2023-03-24 01:55:14,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:14,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:15,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/474b0684e40043a381339a91231a72b7 2023-03-24 01:55:15,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/66fc8a0e34c44a9cb24f7a9e9293cb98 2023-03-24 01:55:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:16,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=600 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/293d65137648445bbf353cca2c3ab659 2023-03-24 01:55:16,080 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/A/474b0684e40043a381339a91231a72b7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/474b0684e40043a381339a91231a72b7 2023-03-24 01:55:16,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/474b0684e40043a381339a91231a72b7, entries=100, sequenceid=600, filesize=9.5 K 2023-03-24 01:55:16,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/B/66fc8a0e34c44a9cb24f7a9e9293cb98 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/66fc8a0e34c44a9cb24f7a9e9293cb98 2023-03-24 01:55:16,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/66fc8a0e34c44a9cb24f7a9e9293cb98, entries=100, sequenceid=600, filesize=9.5 K 2023-03-24 01:55:16,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/.tmp/C/293d65137648445bbf353cca2c3ab659 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/293d65137648445bbf353cca2c3ab659 2023-03-24 01:55:16,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/293d65137648445bbf353cca2c3ab659, entries=100, sequenceid=600, filesize=9.5 K 2023-03-24 01:55:16,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d337c30c8fcf8ea343af6ac65f0aa65a in 1238ms, sequenceid=600, compaction requested=true 2023-03-24 01:55:16,111 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/180e30fc039445109ddfd97835fb180b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8d71fd8e092a4642a1bd32c331202be3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1ba1b199dc29400e9cc8768342c699c6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8051834575e24acc81552d76f75808ec, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/fb8920baf17b4c3c8163c325584c43e7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/26490e5464454b36bc6046b507371ed6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/2d1312a24d1149818dc14a87bd43b83e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/0c2ad3d4bedd456fa3929180cd9c5171, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/375c2fc18538462084df4278abdb871f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814] to archive 2023-03-24 01:55:16,112 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:16,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/84f52e4e459644afa93f5fb654f0f576 2023-03-24 01:55:16,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/59ea8e08df4f40c48298c3b3c7850747 2023-03-24 01:55:16,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/180e30fc039445109ddfd97835fb180b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/180e30fc039445109ddfd97835fb180b 2023-03-24 01:55:16,117 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/6cd9adc4e8934ce3b0bd48496c57dc3c 2023-03-24 01:55:16,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/4e11dc8e0ae4425d9ffd17d661327156 2023-03-24 01:55:16,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8d71fd8e092a4642a1bd32c331202be3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8d71fd8e092a4642a1bd32c331202be3 2023-03-24 01:55:16,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a0569e8c95a42a0a2ddb01a57bd02f6 2023-03-24 01:55:16,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5f325fc86cb4fdf8612dd02b7e38683 2023-03-24 01:55:16,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/5a638f19c1dd477cb3d8f700eae721e4 2023-03-24 01:55:16,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1ba1b199dc29400e9cc8768342c699c6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1ba1b199dc29400e9cc8768342c699c6 2023-03-24 01:55:16,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/20f3a705c06844f1b572ad1703587428 2023-03-24 01:55:16,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/dbecaece114249f7a2f3bb6c3524e484 2023-03-24 01:55:16,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/f340cb89eb82421c84fb8232f736e25b 2023-03-24 01:55:16,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/47e4b0f11c4d4dd691e8faff9cbd734b 2023-03-24 01:55:16,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8051834575e24acc81552d76f75808ec to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8051834575e24acc81552d76f75808ec 2023-03-24 01:55:16,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/b44c15bffeef47e68f29a926f3ff9a11 2023-03-24 01:55:16,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/794b144d3ad94c518090bd51fa2b21f4 2023-03-24 01:55:16,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d085615da5b340f6808636dacfc01a1e 2023-03-24 01:55:16,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/fb8920baf17b4c3c8163c325584c43e7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/fb8920baf17b4c3c8163c325584c43e7 2023-03-24 01:55:16,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/cfb6c6eb7e6c4f24864e796aaa0d8bbb 2023-03-24 01:55:16,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/392124cb5a874d50b49ad18fac15c07b 2023-03-24 01:55:16,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/8f10ea0cfd03419fb0c26621dfde670d 2023-03-24 01:55:16,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/26490e5464454b36bc6046b507371ed6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/26490e5464454b36bc6046b507371ed6 2023-03-24 01:55:16,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/31f38ce434364985ab96a20c4c709cfc 2023-03-24 01:55:16,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/aac7c844a8a64a92b65f98ffc2689d40 2023-03-24 01:55:16,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/2d1312a24d1149818dc14a87bd43b83e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/2d1312a24d1149818dc14a87bd43b83e 2023-03-24 01:55:16,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/7a4e89559eb8421b93c71a91de306ff3 2023-03-24 01:55:16,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/89c05ed7144b467aae3d31d5c0dd0e76 2023-03-24 01:55:16,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/0c2ad3d4bedd456fa3929180cd9c5171 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/0c2ad3d4bedd456fa3929180cd9c5171 2023-03-24 01:55:16,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/21456253dbef4b70bdeb1f0f19e2b9ca 2023-03-24 01:55:16,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d5902687ac514cb5ae2413b5aef7a4c0 2023-03-24 01:55:16,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/1be24c07e75946f999f5d23b8408e3df 2023-03-24 01:55:16,149 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/d9ab890f5f7743c2bd9f6a03470147eb 2023-03-24 01:55:16,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/375c2fc18538462084df4278abdb871f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/375c2fc18538462084df4278abdb871f 2023-03-24 01:55:16,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/77d6b18c8fd042c28345c8b3fa031319 2023-03-24 01:55:16,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/86d3e7acfcad40f694a5f5e4898fe814 2023-03-24 01:55:16,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/eeb25520992341bc97ce4bbb0789b8b1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e79ca3133134ea794549e3e74183481, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/293e24c2c5d34ee2bd4275b2a8526d6d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b16e449d37142e58c0747450149a8f8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2768075d11874ecebfe0e86610f92f7e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31572fc002094f46ac28add3e3dda30a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/70b59cd96d544590a7689ca191c7915b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/5154477d4f694837952eb7c178568ff5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/c9627ded7e034712a1d365fa0411159a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61] to archive 2023-03-24 01:55:16,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:16,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/0d1bfaa4fbaf489a99a2a3f36fac11c2 2023-03-24 01:55:16,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/396fa83616ee465c96602d3fd3f7a61d 2023-03-24 01:55:16,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/eeb25520992341bc97ce4bbb0789b8b1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/eeb25520992341bc97ce4bbb0789b8b1 2023-03-24 01:55:16,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3b3125186144bc6834a8b999066b499 2023-03-24 01:55:16,163 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b613a77905f40e8a69af75cfc7a973e 2023-03-24 01:55:16,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e79ca3133134ea794549e3e74183481 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e79ca3133134ea794549e3e74183481 2023-03-24 01:55:16,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/61d1444ab0c54f55a738085a10161b8f 2023-03-24 01:55:16,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3c50504a49e4444bd8e62efe40d8580 2023-03-24 01:55:16,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/293e24c2c5d34ee2bd4275b2a8526d6d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/293e24c2c5d34ee2bd4275b2a8526d6d 2023-03-24 01:55:16,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a6a64f6d7ee44d5ebacc7e87e6e1aa32 2023-03-24 01:55:16,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/6234e2a489654728a5d00e2a19dcc29b 2023-03-24 01:55:16,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aa94c84733fc413180744793b76b2c0b 2023-03-24 01:55:16,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e9f6045c839a421db18a8ccaf7ac24e4 2023-03-24 01:55:16,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ba5147aa88b24238bdc9e5e5ebda6e73 2023-03-24 01:55:16,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b16e449d37142e58c0747450149a8f8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/8b16e449d37142e58c0747450149a8f8 2023-03-24 01:55:16,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2bef40bba2a5464bab722323e661a1b4 2023-03-24 01:55:16,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f2d11b00680343b4b393360157ec65db 2023-03-24 01:55:16,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2768075d11874ecebfe0e86610f92f7e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/2768075d11874ecebfe0e86610f92f7e 2023-03-24 01:55:16,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/b5e689dbf19144258a508b84ed0bc0ab 2023-03-24 01:55:16,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/459a4630acbb4eb89d16ebecc4f9d275 2023-03-24 01:55:16,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/f6f7188eabe94c37b4f6cb25065c22e8 2023-03-24 01:55:16,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/4e9a8ac45b4448fdbb0050d9c239c2c4 2023-03-24 01:55:16,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31572fc002094f46ac28add3e3dda30a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31572fc002094f46ac28add3e3dda30a 2023-03-24 01:55:16,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/a3f518504f7a49cab30654bd1b5a191f 2023-03-24 01:55:16,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/1efd1b755f3f4b04bc44631b9c7843d1 2023-03-24 01:55:16,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/70b59cd96d544590a7689ca191c7915b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/70b59cd96d544590a7689ca191c7915b 2023-03-24 01:55:16,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/31903cd1d6214a7ba18d1dafc27dd52e 2023-03-24 01:55:16,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/96d7ebce865b448e96bdd79ec4d35989 2023-03-24 01:55:16,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/5154477d4f694837952eb7c178568ff5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/5154477d4f694837952eb7c178568ff5 2023-03-24 01:55:16,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/aafb0ecce7cf402ea880ad700b02c62b 2023-03-24 01:55:16,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/e3eea1ef01be4569a5dec713db5c7846 2023-03-24 01:55:16,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/68e3f70983e54277b8ec20ad757ef717 2023-03-24 01:55:16,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/c9627ded7e034712a1d365fa0411159a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/c9627ded7e034712a1d365fa0411159a 2023-03-24 01:55:16,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/dc09a22d153a4571ad9fd918f979374a 2023-03-24 01:55:16,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/654335e783ef4a3595bdec2f570414de 2023-03-24 01:55:16,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/ef4ff81b867642aba85c32f9caa21f61 2023-03-24 01:55:16,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6c74a1de664c438baffba066872a3ecb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/3a803cd2db4542b3a5a9e7e315d3bf86, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/234ee3adc208446cbaaef33dfab9748c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5baa89539eef4a8ea2dfab656f2d939d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b34cd738f92478897fb875c70f1f978, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4e7c11d35ae04159bb5bb29a11f244f1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1e60b4dd1bfc45f29d947052c8bb3b1b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/360ef6626ac34c149800b8996141711b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/794b6bbcfe5846c5b272ddf77c711fb8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6] to archive 2023-03-24 01:55:16,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:16,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6e07f69df56b4c74b5d24d9aa0626313 2023-03-24 01:55:16,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/d4a77986dd71467e904a0330213ed4e0 2023-03-24 01:55:16,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6c74a1de664c438baffba066872a3ecb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6c74a1de664c438baffba066872a3ecb 2023-03-24 01:55:16,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ce01d1ac67d416a97adf42cb8da9c6a 2023-03-24 01:55:16,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ad50519edb374363be6a6e4c5af6203d 2023-03-24 01:55:16,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/3a803cd2db4542b3a5a9e7e315d3bf86 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/3a803cd2db4542b3a5a9e7e315d3bf86 2023-03-24 01:55:16,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/58fd531cabaa4cc9aa553f9bbc33a702 2023-03-24 01:55:16,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6a3a5c1c1a38464e96cde7e1d52aca95 2023-03-24 01:55:16,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/234ee3adc208446cbaaef33dfab9748c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/234ee3adc208446cbaaef33dfab9748c 2023-03-24 01:55:16,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4bf18bc9066b4aafb678856ae20c2eea 2023-03-24 01:55:16,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/c5854221a55248b791623bdaeb3bcf1a 2023-03-24 01:55:16,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5ba0866736924f62b5f54faa1bbc8e7a 2023-03-24 01:55:16,209 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/bd43257641204d44b0c9fa8d8c6388ff 2023-03-24 01:55:16,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/2979c3f5775c453a801292653ec9c07f 2023-03-24 01:55:16,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5baa89539eef4a8ea2dfab656f2d939d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/5baa89539eef4a8ea2dfab656f2d939d 2023-03-24 01:55:16,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/a640604824fd4ca9891dad79cc27dc4b 2023-03-24 01:55:16,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/69df16e8c66d4d7a84c5d06f9f17bf36 2023-03-24 01:55:16,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b34cd738f92478897fb875c70f1f978 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b34cd738f92478897fb875c70f1f978 2023-03-24 01:55:16,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e40610656456498ba3cc5a0c751198b3 2023-03-24 01:55:16,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6d929fabb05b4aaabfdf359af6dc6476 2023-03-24 01:55:16,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/6900c8fbc18f4ec4a59d07740a0a2fc6 2023-03-24 01:55:16,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/8e193899ce234c7eaff47c6ef72ad87c 2023-03-24 01:55:16,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4e7c11d35ae04159bb5bb29a11f244f1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/4e7c11d35ae04159bb5bb29a11f244f1 2023-03-24 01:55:16,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/10e9139898464ab0aa247b938f0ed408 2023-03-24 01:55:16,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/35bdf9f920d04cc4aa4cddb98cb5f1f3 2023-03-24 01:55:16,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1e60b4dd1bfc45f29d947052c8bb3b1b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1e60b4dd1bfc45f29d947052c8bb3b1b 2023-03-24 01:55:16,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/407ccc1b71e24868a8068be685a0453f 2023-03-24 01:55:16,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/915a8e99784a4b8e9fafb18c0fcbbd32 2023-03-24 01:55:16,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/360ef6626ac34c149800b8996141711b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/360ef6626ac34c149800b8996141711b 2023-03-24 01:55:16,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/9b2ed388379a43339d1e6b718d005e61 2023-03-24 01:55:16,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/1853ea726e884fa5b4a30b60b126fd7a 2023-03-24 01:55:16,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/309cf79bc8d24a0baf204720158e30ed 2023-03-24 01:55:16,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/794b6bbcfe5846c5b272ddf77c711fb8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/794b6bbcfe5846c5b272ddf77c711fb8 2023-03-24 01:55:16,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/09d92ad520c64169bf71d24acd25a059 2023-03-24 01:55:16,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/ce403f118046459a89acdf8d88da8260 2023-03-24 01:55:16,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/e20051c2125647eba6e2bd407c39e5c6 2023-03-24 01:55:16,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/recovered.edits/603.seqid, newMaxSeqId=603, maxSeqId=1 2023-03-24 01:55:16,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a. 2023-03-24 01:55:16,236 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d337c30c8fcf8ea343af6ac65f0aa65a: 2023-03-24 01:55:16,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:16,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d337c30c8fcf8ea343af6ac65f0aa65a, regionState=CLOSED 2023-03-24 01:55:16,238 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622916238"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622916238"}]},"ts":"1679622916238"} 2023-03-24 01:55:16,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-24 01:55:16,240 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure d337c30c8fcf8ea343af6ac65f0aa65a, server=jenkins-hbase4.apache.org,44439,1679622817494 in 2.2490 sec 2023-03-24 01:55:16,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-24 01:55:16,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d337c30c8fcf8ea343af6ac65f0aa65a, UNASSIGN in 2.2540 sec 2023-03-24 01:55:16,241 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622916241"}]},"ts":"1679622916241"} 2023-03-24 01:55:16,242 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:55:16,243 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:55:16,245 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.3600 sec 2023-03-24 01:55:17,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 01:55:17,993 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-24 01:55:17,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:55:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:17,996 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 01:55:17,996 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:18,001 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:18,002 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/recovered.edits] 2023-03-24 01:55:18,005 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/474b0684e40043a381339a91231a72b7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/474b0684e40043a381339a91231a72b7 2023-03-24 01:55:18,006 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/72c2741d96944297bc243d1b3ed3122a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/72c2741d96944297bc243d1b3ed3122a 2023-03-24 01:55:18,006 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/ebcd4f6d671741a2adc812efc23bad2c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/A/ebcd4f6d671741a2adc812efc23bad2c 2023-03-24 01:55:18,009 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/66fc8a0e34c44a9cb24f7a9e9293cb98 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/66fc8a0e34c44a9cb24f7a9e9293cb98 2023-03-24 01:55:18,010 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880529893eaa456cabf675881c68c814 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880529893eaa456cabf675881c68c814 2023-03-24 01:55:18,011 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880633672ff44f2e9c7f994040265129 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/B/880633672ff44f2e9c7f994040265129 2023-03-24 01:55:18,013 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/293d65137648445bbf353cca2c3ab659 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/293d65137648445bbf353cca2c3ab659 2023-03-24 01:55:18,014 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/38eed80928e44bd296e83dd1457ef389 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/38eed80928e44bd296e83dd1457ef389 2023-03-24 01:55:18,015 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/b64f0a32f576442fb8f6ec5f8ae3a634 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/C/b64f0a32f576442fb8f6ec5f8ae3a634 2023-03-24 01:55:18,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/recovered.edits/603.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a/recovered.edits/603.seqid 2023-03-24 01:55:18,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/d337c30c8fcf8ea343af6ac65f0aa65a 2023-03-24 01:55:18,017 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:55:18,019 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:18,020 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:55:18,022 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:55:18,023 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:18,023 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:55:18,023 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679622918023"}]},"ts":"9223372036854775807"} 2023-03-24 01:55:18,024 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:55:18,024 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d337c30c8fcf8ea343af6ac65f0aa65a, NAME => 'TestAcidGuarantees,,1679622890612.d337c30c8fcf8ea343af6ac65f0aa65a.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:55:18,024 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:55:18,024 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679622918024"}]},"ts":"9223372036854775807"} 2023-03-24 01:55:18,025 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:55:18,027 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:18,027 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-03-24 01:55:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 01:55:18,097 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-24 01:55:18,109 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 224) - Thread LEAK? -, OpenFileDescriptor=563 (was 523) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=300 (was 283) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=5415 (was 5596) 2023-03-24 01:55:18,120 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=300, ProcessCount=172, AvailableMemoryMB=5414 2023-03-24 01:55:18,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:55:18,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:55:18,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:18,124 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:55:18,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-24 01:55:18,125 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:55:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:18,126 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:18,126 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f empty. 2023-03-24 01:55:18,127 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:18,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:55:18,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:18,534 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:55:18,535 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a5a2c70f4dec9fd5af77e99de4acd24f, NAME => 'TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:55:18,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a5a2c70f4dec9fd5af77e99de4acd24f, disabling compactions & flushes 2023-03-24 01:55:18,941 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. after waiting 0 ms 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:18,941 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:18,941 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:18,943 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:55:18,944 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622918944"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622918944"}]},"ts":"1679622918944"} 2023-03-24 01:55:18,945 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:55:18,946 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:55:18,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622918946"}]},"ts":"1679622918946"} 2023-03-24 01:55:18,946 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:55:18,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, ASSIGN}] 2023-03-24 01:55:18,951 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, ASSIGN 2023-03-24 01:55:18,952 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:55:19,103 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:19,103 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622919103"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622919103"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622919103"}]},"ts":"1679622919103"} 2023-03-24 01:55:19,104 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:19,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:19,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:19,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a5a2c70f4dec9fd5af77e99de4acd24f, NAME => 'TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:55:19,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:55:19,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,260 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,261 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A 2023-03-24 01:55:19,261 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A 2023-03-24 01:55:19,262 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:19,262 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName A 2023-03-24 01:55:19,262 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:19,262 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,263 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B 2023-03-24 01:55:19,263 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B 2023-03-24 01:55:19,264 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:19,264 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName B 2023-03-24 01:55:19,264 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:19,264 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,265 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C 2023-03-24 01:55:19,265 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C 2023-03-24 01:55:19,265 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:19,265 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName C 2023-03-24 01:55:19,266 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:19,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:19,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:55:19,269 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:19,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:55:19,271 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a5a2c70f4dec9fd5af77e99de4acd24f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73198932, jitterRate=0.09074908494949341}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:55:19,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:19,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., pid=37, masterSystemTime=1679622919255 2023-03-24 01:55:19,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:19,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:19,273 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:19,273 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622919273"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622919273"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622919273"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622919273"}]},"ts":"1679622919273"} 2023-03-24 01:55:19,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-24 01:55:19,275 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 in 170 msec 2023-03-24 01:55:19,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-24 01:55:19,276 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, ASSIGN in 325 msec 2023-03-24 01:55:19,277 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:55:19,277 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622919277"}]},"ts":"1679622919277"} 2023-03-24 01:55:19,277 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:55:19,279 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:55:19,280 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-24 01:55:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 01:55:20,229 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-24 01:55:20,230 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:20,235 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:20,236 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:20,238 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:20,239 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:55:20,240 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52642, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:55:20,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:55:20,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:55:20,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:20,651 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:55:20,652 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:55:20,652 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:55:20,654 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 01:55:20,656 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, REOPEN/MOVE}] 2023-03-24 01:55:20,656 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, REOPEN/MOVE 2023-03-24 01:55:20,657 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:20,657 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622920657"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622920657"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622920657"}]},"ts":"1679622920657"} 2023-03-24 01:55:20,658 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:20,809 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:20,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a5a2c70f4dec9fd5af77e99de4acd24f, disabling compactions & flushes 2023-03-24 01:55:20,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:20,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:20,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. after waiting 0 ms 2023-03-24 01:55:20,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:20,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 01:55:20,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:20,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:20,814 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a5a2c70f4dec9fd5af77e99de4acd24f to self. 2023-03-24 01:55:20,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:20,815 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=CLOSED 2023-03-24 01:55:20,816 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622920815"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622920815"}]},"ts":"1679622920815"} 2023-03-24 01:55:20,817 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-24 01:55:20,817 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 in 158 msec 2023-03-24 01:55:20,818 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=true 2023-03-24 01:55:20,968 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:20,968 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622920968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622920968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622920968"}]},"ts":"1679622920968"} 2023-03-24 01:55:20,969 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:21,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a5a2c70f4dec9fd5af77e99de4acd24f, NAME => 'TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:55:21,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:55:21,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,125 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,126 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A 2023-03-24 01:55:21,126 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A 2023-03-24 01:55:21,127 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:21,127 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName A 2023-03-24 01:55:21,128 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:21,128 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,129 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B 2023-03-24 01:55:21,129 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B 2023-03-24 01:55:21,129 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:21,129 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName B 2023-03-24 01:55:21,130 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:21,130 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,130 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C 2023-03-24 01:55:21,130 DEBUG [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C 2023-03-24 01:55:21,130 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:21,131 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5a2c70f4dec9fd5af77e99de4acd24f columnFamilyName C 2023-03-24 01:55:21,131 INFO [StoreOpener-a5a2c70f4dec9fd5af77e99de4acd24f-1] regionserver.HStore(310): Store=a5a2c70f4dec9fd5af77e99de4acd24f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:21,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,134 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:55:21,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a5a2c70f4dec9fd5af77e99de4acd24f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59720117, jitterRate=-0.11010091006755829}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:55:21,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:21,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., pid=42, masterSystemTime=1679622921121 2023-03-24 01:55:21,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,138 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,138 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622921138"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622921138"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622921138"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622921138"}]},"ts":"1679622921138"} 2023-03-24 01:55:21,140 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-24 01:55:21,140 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 in 170 msec 2023-03-24 01:55:21,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-24 01:55:21,141 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, REOPEN/MOVE in 484 msec 2023-03-24 01:55:21,142 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-24 01:55:21,142 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-03-24 01:55:21,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-03-24 01:55:21,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-24 01:55:21,145 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,149 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,150 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,154 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,154 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,157 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,157 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,160 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,161 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,164 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,164 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,169 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,170 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,175 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,176 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,180 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,181 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,184 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,184 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:21,187 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:21,194 DEBUG [hconnection-0x1db04254-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,194 DEBUG [hconnection-0x5dc313dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,195 DEBUG [hconnection-0x29015de1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,195 DEBUG [hconnection-0x6d1b98e4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,196 DEBUG [hconnection-0x536a4c39-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,196 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,197 DEBUG [hconnection-0x65518729-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,196 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,196 DEBUG [hconnection-0x6f1c59bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,197 DEBUG [hconnection-0x6e6b86cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,197 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56622, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,197 DEBUG [hconnection-0x5f6d1309-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,199 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56618, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,200 DEBUG [hconnection-0x5b0f31ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:21,201 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,202 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56624, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,204 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56664, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,204 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56636, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,205 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56680, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,207 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56690, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,208 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,214 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:21,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:55:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:21,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:21,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:21,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,224 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f52ca3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,235 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,235 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,242 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,242 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622981238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622981238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622981239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622981240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622981244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,260 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244d7b3d0d99ea4c999e7eac2f5aa6dca8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244d7b3d0d99ea4c999e7eac2f5aa6dca8 2023-03-24 01:55:21,342 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:21,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:21,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622981345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622981345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622981348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622981348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622981352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,360 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,363 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,372 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,372 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,372 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,372 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@455aa459[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,377 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,377 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,377 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,378 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,380 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,381 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,385 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,388 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,388 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,388 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,388 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,388 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,389 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,389 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,489 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:21,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,495 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,497 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,497 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:21,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,503 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,503 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,506 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d23551[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,514 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,514 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,514 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,524 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,524 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,526 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,526 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,526 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,526 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,528 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,532 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,533 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,533 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622981548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622981548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622981550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622981550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622981554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,633 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:21,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,637 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,637 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,643 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,647 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d308980[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,649 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,649 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,650 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,650 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,651 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,655 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,655 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,662 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,663 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,663 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,680 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/9ec0c5a0bda5463f9f63933eba2ada77 2023-03-24 01:55:21,763 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:21,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,766 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,768 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,768 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,768 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,768 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,770 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,771 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50cd2ab1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,774 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,776 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,776 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,778 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,779 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,790 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,790 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,790 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,791 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,792 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,797 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,796 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,797 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,797 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,797 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,797 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,797 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622981853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622981853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622981853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622981854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:21,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622981863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,897 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:21,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:21,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:21,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:21,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,904 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,904 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:21,905 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:21,906 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:21,906 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:21,907 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a5349d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:21,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:21,909 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:21,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,912 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:21,914 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:21,914 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:21,914 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:21,914 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:21,914 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:21,917 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,917 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,917 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:21,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,920 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,922 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:21,923 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:21,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:21,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:21,923 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:21,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:21,923 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:21,923 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:21,923 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:21,923 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:22,023 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:22,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:22,026 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:22,026 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:22,026 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:22,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:22,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,027 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,028 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,028 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:22,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:22,028 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,028 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:22,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:22,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:22,030 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:22,030 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:22,030 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:22,030 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:22,030 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:22,036 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:22,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,036 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7865f965[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:22,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:22,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:22,040 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:22,043 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:22,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:22,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:22,043 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:22,043 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:22,046 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,046 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:22,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:22,050 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:22,050 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:22,050 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:22,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:22,050 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:22,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:22,050 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:22,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:22,051 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,051 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:22,051 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:22,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/9b0e4e933b52412d8023dbcbb2059816 2023-03-24 01:55:22,099 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/5ce915cfcb564991b6d64543d91f7ae4 2023-03-24 01:55:22,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/9ec0c5a0bda5463f9f63933eba2ada77 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77 2023-03-24 01:55:22,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77, entries=150, sequenceid=18, filesize=25.1 K 2023-03-24 01:55:22,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/9b0e4e933b52412d8023dbcbb2059816 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816 2023-03-24 01:55:22,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816, entries=150, sequenceid=18, filesize=11.6 K 2023-03-24 01:55:22,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/5ce915cfcb564991b6d64543d91f7ae4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4 2023-03-24 01:55:22,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4, entries=150, sequenceid=18, filesize=11.6 K 2023-03-24 01:55:22,121 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a5a2c70f4dec9fd5af77e99de4acd24f in 900ms, sequenceid=18, compaction requested=false 2023-03-24 01:55:22,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:22,151 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:22,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:22,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:22,154 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,155 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,156 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,156 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:22,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:22,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:22,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:22,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:22,158 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:22,158 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:22,158 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:22,158 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:22,159 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:22,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622982398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622982399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622982400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622982400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622982403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622982506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622982506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622982506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622982506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622982511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,573 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242af61bb2960b4c6bb92fdfc08de74ec8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242af61bb2960b4c6bb92fdfc08de74ec8 2023-03-24 01:55:22,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622982710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622982711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622982711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622982712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:22,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622982716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:22,979 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/68518042d6be4807bba7937abfaefb73 2023-03-24 01:55:22,994 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/9393885984974927bef8353ab580b72a 2023-03-24 01:55:23,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622983015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622983016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622983016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622983017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622983023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,423 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/2dd1d65a63974fbab3a152c419df306b 2023-03-24 01:55:23,427 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/68518042d6be4807bba7937abfaefb73 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73 2023-03-24 01:55:23,437 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73, entries=150, sequenceid=36, filesize=25.1 K 2023-03-24 01:55:23,439 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/9393885984974927bef8353ab580b72a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a 2023-03-24 01:55:23,443 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a, entries=150, sequenceid=36, filesize=11.6 K 2023-03-24 01:55:23,444 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/2dd1d65a63974fbab3a152c419df306b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b 2023-03-24 01:55:23,448 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b, entries=150, sequenceid=36, filesize=11.6 K 2023-03-24 01:55:23,449 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a5a2c70f4dec9fd5af77e99de4acd24f in 1291ms, sequenceid=36, compaction requested=false 2023-03-24 01:55:23,449 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:23,449 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:23,451 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,451 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:23,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41a01360[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:23,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:23,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,456 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:23,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:23,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:23,462 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:23,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:23,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:23,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:23,462 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:23,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:23,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:23,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,472 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:23,472 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:23,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:23,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:23,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622983541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622983542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622983543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622983550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622983551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,572 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:23,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:23,575 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:23,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:23,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:23,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,576 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:23,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:23,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:23,578 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,578 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:23,578 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:23,578 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:23,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:23,579 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:23,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78012147[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:23,581 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:23,581 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:23,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:23,585 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:23,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:23,587 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:23,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:23,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:23,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:23,587 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:23,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:23,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:23,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,593 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:23,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,593 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:23,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:23,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622983651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622983651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622983651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622983654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622983662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,693 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:23,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:23,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:23,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:23,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:23,696 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:23,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:23,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:23,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:23,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:23,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:23,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:23,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,703 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:23,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1169f306[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:23,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:23,705 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:23,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:23,711 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,711 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:23,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:23,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:23,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:23,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:23,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:23,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:23,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:23,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,720 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:23,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,720 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,820 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:23,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:23,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:23,823 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,825 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:23,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:23,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:23,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:23,827 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:23,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,829 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,829 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:23,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46108e9f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:23,830 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:23,831 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,832 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:23,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:23,833 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,833 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:23,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:23,836 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:23,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:23,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:23,836 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:23,836 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:23,841 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,841 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,841 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,842 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:23,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:23,849 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:23,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,849 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,849 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622983857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622983857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622983858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622983858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622983868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,944 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245958be374b86495087a02342d80cfd60 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245958be374b86495087a02342d80cfd60 2023-03-24 01:55:23,949 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:23,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:23,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:23,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:23,952 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:23,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:23,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:23,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:23,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:23,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:23,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:23,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:23,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:23,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:23,957 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,957 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76abe2e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:23,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:23,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,962 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:23,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:23,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,964 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:23,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:23,966 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:23,966 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:23,966 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:23,966 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:23,966 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:23,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:23,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:23,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:23,970 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:23,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:23,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:23,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:23,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:23,972 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:23,972 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:23,972 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,072 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:24,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:24,075 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:24,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:24,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:24,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:24,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:24,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:24,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:24,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:24,080 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:24,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:24,082 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:24,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7610bc9d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:24,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:24,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:24,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:24,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:24,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:24,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:24,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:24,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:24,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:24,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:24,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:24,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:24,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:24,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:24,098 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:24,098 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:24,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622984159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622984161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622984165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622984166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622984174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,198 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:24,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:24,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:24,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:24,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:24,201 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:24,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:24,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:24,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:24,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:24,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:24,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:24,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:24,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,208 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:24,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28bf909a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:24,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:24,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:24,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:24,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:24,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:24,214 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:24,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,214 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:24,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:24,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:24,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:24,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:24,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:24,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:24,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:24,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:24,223 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:24,223 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:24,323 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:24,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:24,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:24,326 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:24,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,329 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,329 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,329 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,330 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:24,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:24,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:24,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:24,339 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,339 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:24,339 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:24,339 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:24,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,341 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:24,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a999764[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:24,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:24,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:24,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:24,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:24,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:24,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:24,347 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:24,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:24,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:24,347 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:24,347 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:24,349 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/ef2277e0178f45c78fded6a8263fecd1 2023-03-24 01:55:24,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:24,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:24,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:24,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,356 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:24,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:24,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:24,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,357 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:24,357 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:24,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:24,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:24,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/7d00a1813ce14affbd2bf7f45c149e27 2023-03-24 01:55:24,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a97cadd60ba34f43965872a2e8101165 2023-03-24 01:55:24,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/ef2277e0178f45c78fded6a8263fecd1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1 2023-03-24 01:55:24,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1, entries=200, sequenceid=51, filesize=31.9 K 2023-03-24 01:55:24,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/7d00a1813ce14affbd2bf7f45c149e27 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27 2023-03-24 01:55:24,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 01:55:24,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a97cadd60ba34f43965872a2e8101165 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165 2023-03-24 01:55:24,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 01:55:24,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for a5a2c70f4dec9fd5af77e99de4acd24f in 863ms, sequenceid=51, compaction requested=true 2023-03-24 01:55:24,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:24,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:24,386 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:24,386 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:24,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:24,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:24,387 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:24,387 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:24,387 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:24,387 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:24,387 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,387 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,388 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=82.2 K 2023-03-24 01:55:24,388 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=34.8 K 2023-03-24 01:55:24,389 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9ec0c5a0bda5463f9f63933eba2ada77, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679622921217 2023-03-24 01:55:24,390 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9b0e4e933b52412d8023dbcbb2059816, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679622921217 2023-03-24 01:55:24,390 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 68518042d6be4807bba7937abfaefb73, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679622921236 2023-03-24 01:55:24,390 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9393885984974927bef8353ab580b72a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679622921236 2023-03-24 01:55:24,390 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ef2277e0178f45c78fded6a8263fecd1, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922398 2023-03-24 01:55:24,390 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7d00a1813ce14affbd2bf7f45c149e27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922399 2023-03-24 01:55:24,399 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:24,400 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#315 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:24,457 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:24,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:24,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:24,460 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:24,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:24,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:24,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:24,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:24,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:24,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:24,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:24,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:24,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:24,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:24,464 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,464 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:24,464 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:55:24,464 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:24,464 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:24,464 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:24,465 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:24,465 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:24,465 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:24,479 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242a4097f67dd34429a798c4785b84cfbc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a4097f67dd34429a798c4785b84cfbc 2023-03-24 01:55:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:24,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622984682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622984682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622984688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622984691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622984691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622984793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622984793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622984793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622984800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622984800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:24,812 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/b02f4061280143cfbb68aed6079a2ac0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b02f4061280143cfbb68aed6079a2ac0 2023-03-24 01:55:24,821 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into b02f4061280143cfbb68aed6079a2ac0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:24,821 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:24,821 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=13, startTime=1679622924386; duration=0sec 2023-03-24 01:55:24,821 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:24,821 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:24,827 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:24,827 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:24,827 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:24,828 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=34.8 K 2023-03-24 01:55:24,828 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5ce915cfcb564991b6d64543d91f7ae4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1679622921217 2023-03-24 01:55:24,828 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2dd1d65a63974fbab3a152c419df306b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679622921236 2023-03-24 01:55:24,829 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a97cadd60ba34f43965872a2e8101165, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922399 2023-03-24 01:55:24,835 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:24,885 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/f3c967c49abf46b8a39847f2b47675b6 2023-03-24 01:55:24,899 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/524b8594a8b84e4b87dcc049ee37e2d4 2023-03-24 01:55:25,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622984999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622984999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622985001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622985010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622985010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,235 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/43312bcd6f2b42bb90a631de192c7938 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/43312bcd6f2b42bb90a631de192c7938 2023-03-24 01:55:25,239 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into 43312bcd6f2b42bb90a631de192c7938(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:25,239 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:25,239 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=13, startTime=1679622924386; duration=0sec 2023-03-24 01:55:25,239 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:25,248 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/fdf17f4593954afdb5fc8f7092184df7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/fdf17f4593954afdb5fc8f7092184df7 2023-03-24 01:55:25,252 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into fdf17f4593954afdb5fc8f7092184df7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:25,252 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:25,252 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=13, startTime=1679622924386; duration=0sec 2023-03-24 01:55:25,252 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:25,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622985305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622985305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622985306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,312 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a918c73dceaa48bca676d0e6f436012a 2023-03-24 01:55:25,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/f3c967c49abf46b8a39847f2b47675b6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6 2023-03-24 01:55:25,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622985315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622985317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,320 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6, entries=150, sequenceid=69, filesize=25.1 K 2023-03-24 01:55:25,321 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/524b8594a8b84e4b87dcc049ee37e2d4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4 2023-03-24 01:55:25,324 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4, entries=150, sequenceid=69, filesize=11.6 K 2023-03-24 01:55:25,325 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a918c73dceaa48bca676d0e6f436012a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a 2023-03-24 01:55:25,328 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a, entries=150, sequenceid=69, filesize=11.6 K 2023-03-24 01:55:25,328 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a5a2c70f4dec9fd5af77e99de4acd24f in 864ms, sequenceid=69, compaction requested=false 2023-03-24 01:55:25,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:25,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:25,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:25,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38d0dfd7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:25,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:25,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:25,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:25,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:25,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:25,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:25,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:25,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:25,339 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:25,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:25,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:25,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:25,339 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:25,341 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,341 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:25,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,341 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:25,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:25,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:25,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:25,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:25,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:25,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:25,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:25,353 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:25,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,354 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:25,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:25,454 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:25,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:25,457 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:25,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:25,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:25,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:25,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:25,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:25,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:25,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:25,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:25,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:25,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:25,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:25,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:25,461 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:25,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:25,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:25,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622985828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622985829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622985829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622985837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622985837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,882 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032461508d0964f94d1ebea886e47ceb196e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461508d0964f94d1ebea886e47ceb196e 2023-03-24 01:55:25,888 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/b2c51f50230c43378faea56e0edcf822 2023-03-24 01:55:25,904 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/2a8742d03084452bb3922dd5d88b7a34 2023-03-24 01:55:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622985938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622985939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622985939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622985949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:25,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622985949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622986145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622986145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622986146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622986156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622986157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,318 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/0daba68d37ed49af94ec19cc2c9d7134 2023-03-24 01:55:26,322 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/b2c51f50230c43378faea56e0edcf822 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822 2023-03-24 01:55:26,326 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822, entries=150, sequenceid=86, filesize=25.1 K 2023-03-24 01:55:26,327 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/2a8742d03084452bb3922dd5d88b7a34 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34 2023-03-24 01:55:26,330 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34, entries=150, sequenceid=86, filesize=11.6 K 2023-03-24 01:55:26,331 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/0daba68d37ed49af94ec19cc2c9d7134 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134 2023-03-24 01:55:26,337 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134, entries=150, sequenceid=86, filesize=11.6 K 2023-03-24 01:55:26,338 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for a5a2c70f4dec9fd5af77e99de4acd24f in 877ms, sequenceid=86, compaction requested=true 2023-03-24 01:55:26,338 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:26,338 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,341 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b577804[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,355 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,355 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,365 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:26,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:26,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:26,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,365 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:26,365 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:26,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:26,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:26,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:26,465 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:26,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:26,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:26,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:26,469 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:26,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:26,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:26,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,472 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032415f02af9b8814fc69eda168d70b105ee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032415f02af9b8814fc69eda168d70b105ee 2023-03-24 01:55:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,473 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,473 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:26,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622986466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:26,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f380ab4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622986469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622986470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622986472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622986474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,485 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:26,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,491 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:26,491 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,491 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:26,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:26,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622986574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622986580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622986580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622986581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622986583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,592 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:26,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:26,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:26,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:26,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:26,595 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:26,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:26,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:26,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:26,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:26,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:26,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:26,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:26,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393c9c80[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,610 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,614 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:26,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:26,617 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:26,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,618 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:26,618 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:26,718 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:26,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:26,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:26,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:26,721 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:26,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:26,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:26,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:26,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:26,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:26,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:26,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:26,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:26,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:26,729 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,729 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:26,729 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:26,729 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7acd217[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,733 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,737 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,737 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,742 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:26,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,745 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:26,745 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622986780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622986788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622986788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622986788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:26,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622986789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,845 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:26,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:26,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:26,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:26,848 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:26,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:26,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:26,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:26,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:26,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:26,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:26,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:26,854 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:26,854 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,854 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,855 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cb0dbe[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,862 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,863 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,866 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,866 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:26,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:26,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,872 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:26,872 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:26,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:26,879 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=106, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/c102b9ef1dfc4272b79f2f1d0abddca2 2023-03-24 01:55:26,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/99c75f6bb74a46e19cf8b6e301be359e 2023-03-24 01:55:26,972 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:26,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:26,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:26,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:26,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:26,975 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:26,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:26,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:26,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,983 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:26,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:26,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:26,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:26,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:26,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:26,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:26,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:26,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:26,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:26,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:26,985 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,985 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:26,985 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:26,985 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:26,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2879c459[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:26,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:26,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:26,990 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:26,990 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:26,990 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:26,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:26,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:26,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:26,994 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:26,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:26,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:26,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:26,994 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:26,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:26,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:26,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:27,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:27,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,001 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:27,001 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:27,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622987087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622987092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622987092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622987093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,101 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:27,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622987096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:27,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:27,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:27,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:27,104 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:27,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:27,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:27,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,105 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:27,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:27,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:27,108 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,108 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:27,108 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:27,108 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:27,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:27,113 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,114 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,114 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:27,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757c954a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:27,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:27,118 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:27,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:27,121 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:27,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:27,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:27,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:27,121 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:27,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,123 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:27,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:27,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,127 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,127 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,128 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:27,128 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,128 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:27,128 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,128 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,128 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:27,228 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:27,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:27,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:27,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:27,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:27,231 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:27,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:27,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:27,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,233 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:27,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:27,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:27,234 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:27,235 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,235 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:27,235 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:27,235 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:27,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:27,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a1cd98[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:27,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:27,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:27,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:27,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:27,244 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:27,245 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:27,245 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:27,245 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:27,245 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:27,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,247 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:27,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,250 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:27,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,250 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d6d4a66e63504c79a6d9431bac0d65b5 2023-03-24 01:55:27,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/c102b9ef1dfc4272b79f2f1d0abddca2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2 2023-03-24 01:55:27,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2, entries=200, sequenceid=106, filesize=31.9 K 2023-03-24 01:55:27,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/99c75f6bb74a46e19cf8b6e301be359e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e 2023-03-24 01:55:27,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e, entries=150, sequenceid=106, filesize=11.6 K 2023-03-24 01:55:27,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d6d4a66e63504c79a6d9431bac0d65b5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5 2023-03-24 01:55:27,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5, entries=150, sequenceid=106, filesize=11.6 K 2023-03-24 01:55:27,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a5a2c70f4dec9fd5af77e99de4acd24f in 863ms, sequenceid=106, compaction requested=true 2023-03-24 01:55:27,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:27,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:27,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:27,319 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:27,319 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:27,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:27,320 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:27,320 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:27,320 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:27,320 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,320 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:27,321 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,321 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/43312bcd6f2b42bb90a631de192c7938, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=107.4 K 2023-03-24 01:55:27,321 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b02f4061280143cfbb68aed6079a2ac0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=46.5 K 2023-03-24 01:55:27,321 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 43312bcd6f2b42bb90a631de192c7938, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922399 2023-03-24 01:55:27,321 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b02f4061280143cfbb68aed6079a2ac0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922399 2023-03-24 01:55:27,321 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f3c967c49abf46b8a39847f2b47675b6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679622923538 2023-03-24 01:55:27,321 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 524b8594a8b84e4b87dcc049ee37e2d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679622923538 2023-03-24 01:55:27,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b2c51f50230c43378faea56e0edcf822, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679622924681 2023-03-24 01:55:27,322 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2a8742d03084452bb3922dd5d88b7a34, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679622924681 2023-03-24 01:55:27,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c102b9ef1dfc4272b79f2f1d0abddca2, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:27,322 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 99c75f6bb74a46e19cf8b6e301be359e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:27,330 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:27,333 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#328 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:27,351 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:27,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:27,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:27,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:27,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:27,355 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:27,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:27,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:27,356 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,356 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:27,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:27,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:27,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:27,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:27,364 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:27,364 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:27,365 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:27,376 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249404401014694b3ca0aee6f2a3efe85a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249404401014694b3ca0aee6f2a3efe85a 2023-03-24 01:55:27,380 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/330991ff60714c0e875bea7f6e5bd365 2023-03-24 01:55:27,389 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/c8ad3316900c4ef5bb720382d0c914aa 2023-03-24 01:55:27,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:27,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:27,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622987616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622987616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622987617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622987619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622987622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622987722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622987725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622987725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622987725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622987728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,742 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/701edbfedef04eb8a087f99358e929e8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/701edbfedef04eb8a087f99358e929e8 2023-03-24 01:55:27,746 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into 701edbfedef04eb8a087f99358e929e8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:27,746 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:27,746 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=12, startTime=1679622927319; duration=0sec 2023-03-24 01:55:27,746 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:27,746 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:27,747 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:27,747 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:27,747 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,747 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/fdf17f4593954afdb5fc8f7092184df7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=46.5 K 2023-03-24 01:55:27,747 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting fdf17f4593954afdb5fc8f7092184df7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622922399 2023-03-24 01:55:27,748 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a918c73dceaa48bca676d0e6f436012a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679622923538 2023-03-24 01:55:27,748 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0daba68d37ed49af94ec19cc2c9d7134, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679622924681 2023-03-24 01:55:27,748 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting d6d4a66e63504c79a6d9431bac0d65b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:27,754 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:27,769 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/4f88b93142824e849a09498c48bd0d48 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4f88b93142824e849a09498c48bd0d48 2023-03-24 01:55:27,772 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into 4f88b93142824e849a09498c48bd0d48(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:27,772 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:27,772 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=12, startTime=1679622927319; duration=0sec 2023-03-24 01:55:27,772 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:27,803 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/cf18196a6360442b99eeab037f7146df 2023-03-24 01:55:27,807 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/330991ff60714c0e875bea7f6e5bd365 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365 2023-03-24 01:55:27,810 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365, entries=150, sequenceid=118, filesize=25.1 K 2023-03-24 01:55:27,811 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/c8ad3316900c4ef5bb720382d0c914aa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa 2023-03-24 01:55:27,814 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa, entries=150, sequenceid=118, filesize=11.6 K 2023-03-24 01:55:27,815 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/cf18196a6360442b99eeab037f7146df as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df 2023-03-24 01:55:27,818 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df, entries=150, sequenceid=118, filesize=11.6 K 2023-03-24 01:55:27,818 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a5a2c70f4dec9fd5af77e99de4acd24f in 454ms, sequenceid=118, compaction requested=false 2023-03-24 01:55:27,818 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:27,818 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,818 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:27,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:27,820 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:27,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b28844f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:27,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:27,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:27,823 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:27,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:27,828 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:27,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:27,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:27,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:27,829 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:27,833 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,833 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,833 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:27,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:27,839 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:27,839 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:27,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:27,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:27,940 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:27,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:27,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:27,943 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,945 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622987941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:27,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:27,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:27,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:27,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:27,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:27,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,948 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d15c47f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:27,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:27,950 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622987942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622987942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622987943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:27,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:27,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:27,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622987945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:27,956 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:27,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:27,958 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:27,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:27,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:27,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:27,958 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:27,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:27,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:27,960 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:27,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:27,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:27,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:27,965 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:27,965 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:27,965 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:27,965 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:27,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622988046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622988051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622988051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622988051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622988053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,065 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,068 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c52cc52[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,081 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,081 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,087 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,087 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,087 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,162 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/cc97830cd3ea4aeaa1a3a9180b444605 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cc97830cd3ea4aeaa1a3a9180b444605 2023-03-24 01:55:28,166 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into cc97830cd3ea4aeaa1a3a9180b444605(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:28,166 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:28,166 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=12, startTime=1679622927319; duration=0sec 2023-03-24 01:55:28,167 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:28,187 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,191 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,195 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,197 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,197 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,197 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,197 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5605054d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,203 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,204 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,207 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,207 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,210 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:28,220 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,220 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622988253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622988256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622988256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622988257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622988261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,320 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,325 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,326 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f0e93e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,334 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,340 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,340 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,340 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,343 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303243a1445d1eee647b6af6c679d45aa418e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a1445d1eee647b6af6c679d45aa418e 2023-03-24 01:55:28,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,354 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=141, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/7c4cb8ddeb76480294c79d49f79b1351 2023-03-24 01:55:28,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:28,354 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,354 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,354 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/d004583d9afe4f5ea01e1aaed454ef1d 2023-03-24 01:55:28,454 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,457 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,463 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,463 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,463 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,463 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,464 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab5e643[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,467 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,472 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,472 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,488 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,488 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622988559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622988564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622988566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622988566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622988568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,588 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,591 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,595 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,595 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f811b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,612 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,612 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,712 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,715 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:28,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8ed330[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:28,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:28,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,723 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:28,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:28,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:28,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,726 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:28,728 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:28,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:28,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:28,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:28,728 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:28,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:28,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,730 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,734 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:28,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:28,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:28,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:28,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,735 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:28,735 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:28,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:28,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/7f487cd1ad0649f3936d1459e6c344eb 2023-03-24 01:55:28,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/7c4cb8ddeb76480294c79d49f79b1351 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351 2023-03-24 01:55:28,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351, entries=200, sequenceid=141, filesize=32.1 K 2023-03-24 01:55:28,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/d004583d9afe4f5ea01e1aaed454ef1d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d 2023-03-24 01:55:28,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d, entries=150, sequenceid=141, filesize=11.8 K 2023-03-24 01:55:28,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/7f487cd1ad0649f3936d1459e6c344eb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb 2023-03-24 01:55:28,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb, entries=150, sequenceid=141, filesize=11.8 K 2023-03-24 01:55:28,803 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for a5a2c70f4dec9fd5af77e99de4acd24f in 873ms, sequenceid=141, compaction requested=true 2023-03-24 01:55:28,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:28,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:28,803 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:28,803 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:28,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:28,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:28,804 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:28,804 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:28,804 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:28,804 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:28,805 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,805 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,805 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4f88b93142824e849a09498c48bd0d48, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=82.6 K 2023-03-24 01:55:28,805 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/701edbfedef04eb8a087f99358e929e8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=35.2 K 2023-03-24 01:55:28,805 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4f88b93142824e849a09498c48bd0d48, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:28,805 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 701edbfedef04eb8a087f99358e929e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:28,805 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 330991ff60714c0e875bea7f6e5bd365, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679622926464 2023-03-24 01:55:28,806 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c8ad3316900c4ef5bb720382d0c914aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679622926464 2023-03-24 01:55:28,806 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting d004583d9afe4f5ea01e1aaed454ef1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:28,812 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#336 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:28,812 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7c4cb8ddeb76480294c79d49f79b1351, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:28,833 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#337 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:28,835 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:28,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:28,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:28,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:28,844 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,844 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:28,849 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:28,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:28,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:28,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:28,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:28,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:28,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:28,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:28,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:28,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:28,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:28,850 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:55:28,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:28,851 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:28,851 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:28,851 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:28,851 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:28,851 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:28,868 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d0a6cfd03a3540e985185f0e3d929653 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d0a6cfd03a3540e985185f0e3d929653 2023-03-24 01:55:28,873 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/21f5f964f8064af7b37c7797f984dd4b 2023-03-24 01:55:28,881 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/b87b7a4957534b0ca96451b9e34d8c7a 2023-03-24 01:55:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:29,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:29,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622989095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622929092, max timestamp=1679622929105Num uniques 859; 2023-03-24 01:55:29,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622989097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622929092, max timestamp=1679622929105Num uniques 859; 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622929092, max timestamp=1679622929105Num uniques 859; 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:55:29,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:55:29,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622989141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622989142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622989142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622989209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622989209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,230 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/3b186e7595d6400aa7d352b02e296885 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/3b186e7595d6400aa7d352b02e296885 2023-03-24 01:55:29,233 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into 3b186e7595d6400aa7d352b02e296885(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:29,234 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,234 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=13, startTime=1679622928803; duration=0sec 2023-03-24 01:55:29,234 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:29,234 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:29,235 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:29,235 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:29,235 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,235 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cc97830cd3ea4aeaa1a3a9180b444605, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=35.2 K 2023-03-24 01:55:29,235 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cc97830cd3ea4aeaa1a3a9180b444605, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1679622925825 2023-03-24 01:55:29,235 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cf18196a6360442b99eeab037f7146df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679622926464 2023-03-24 01:55:29,236 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7f487cd1ad0649f3936d1459e6c344eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:29,240 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#341 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:29,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622989243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622989249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622989250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,291 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a115d4086a3547b9a5aa97817a177006 2023-03-24 01:55:29,295 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/21f5f964f8064af7b37c7797f984dd4b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b 2023-03-24 01:55:29,299 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b, entries=150, sequenceid=153, filesize=25.3 K 2023-03-24 01:55:29,300 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/b87b7a4957534b0ca96451b9e34d8c7a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a 2023-03-24 01:55:29,303 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a, entries=150, sequenceid=153, filesize=11.8 K 2023-03-24 01:55:29,304 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/a115d4086a3547b9a5aa97817a177006 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006 2023-03-24 01:55:29,307 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006, entries=150, sequenceid=153, filesize=11.8 K 2023-03-24 01:55:29,307 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a5a2c70f4dec9fd5af77e99de4acd24f in 457ms, sequenceid=153, compaction requested=false 2023-03-24 01:55:29,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:29,309 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,310 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50997cc6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:29,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:29,311 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,312 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:29,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:29,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:29,314 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,314 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:29,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:29,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:29,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:29,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:29,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:29,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,320 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:29,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:29,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,322 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,322 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,323 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:29,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,323 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,323 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:29,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:29,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:29,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:29,420 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:29,423 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:29,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:29,426 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:29,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:29,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:29,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,428 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,428 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:29,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:29,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:29,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:29,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:29,431 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:29,432 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:29,433 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245878230626ee4a62b368daf7b49c8a9c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245878230626ee4a62b368daf7b49c8a9c 2023-03-24 01:55:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25eb1f68[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:29,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:29,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,438 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,438 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/280258a2e3834518ae0340491a2facfd 2023-03-24 01:55:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:29,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:29,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622989439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622989439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:29,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:29,443 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:29,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:29,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:29,443 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:29,443 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:29,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,447 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/db2dcdd7171547679c4d8c471d5a1f02 2023-03-24 01:55:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,449 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,452 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:29,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:29,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,453 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:29,453 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:29,453 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622989450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622989452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622989454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/e2d5baff3dc14714b399a230a62524b9 2023-03-24 01:55:29,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/280258a2e3834518ae0340491a2facfd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd 2023-03-24 01:55:29,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd, entries=200, sequenceid=190, filesize=32.1 K 2023-03-24 01:55:29,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/db2dcdd7171547679c4d8c471d5a1f02 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02 2023-03-24 01:55:29,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02, entries=150, sequenceid=190, filesize=11.8 K 2023-03-24 01:55:29,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/e2d5baff3dc14714b399a230a62524b9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9 2023-03-24 01:55:29,473 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9, entries=150, sequenceid=190, filesize=11.8 K 2023-03-24 01:55:29,474 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for a5a2c70f4dec9fd5af77e99de4acd24f in 55ms, sequenceid=190, compaction requested=true 2023-03-24 01:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:55:29,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 01:55:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:29,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:29,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:29,554 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:29,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:29,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:29,558 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,561 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,562 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:29,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:29,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:29,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:29,568 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,568 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:29,568 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:29,568 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:29,570 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:29,571 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,573 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:29,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c01597[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:29,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:29,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,576 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:29,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:29,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:29,579 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,579 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:29,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:29,583 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:29,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:29,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:29,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:29,583 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:29,584 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032402ce4d2eebae4de79150bd2c9188d37c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032402ce4d2eebae4de79150bd2c9188d37c 2023-03-24 01:55:29,586 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,586 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,586 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,592 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:29,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:29,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,592 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:29,592 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:29,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,598 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/6d42d813a581471eb8ac24e8f6fd5844 2023-03-24 01:55:29,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622929579, max timestamp=1679622929640Num uniques 859; 2023-03-24 01:55:29,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:29,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622929579, max timestamp=1679622929640Num uniques 859; 2023-03-24 01:55:29,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:29,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622929579, max timestamp=1679622929640Num uniques 859; 2023-03-24 01:55:29,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:29,650 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/354fd785fd9942d6b25d295ed79ba45f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/354fd785fd9942d6b25d295ed79ba45f 2023-03-24 01:55:29,658 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into 354fd785fd9942d6b25d295ed79ba45f(size=12.1 K), total size for store is 35.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:29,658 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,658 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=13, startTime=1679622928803; duration=0sec 2023-03-24 01:55:29,658 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 01:55:29,658 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-24 01:55:29,659 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:55:29,659 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:55:29,659 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. because compaction request was cancelled 2023-03-24 01:55:29,659 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:29,660 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:29,661 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:29,661 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,661 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/3b186e7595d6400aa7d352b02e296885, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=35.6 K 2023-03-24 01:55:29,661 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3b186e7595d6400aa7d352b02e296885, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:29,662 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b87b7a4957534b0ca96451b9e34d8c7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679622927940 2023-03-24 01:55:29,663 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting db2dcdd7171547679c4d8c471d5a1f02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679622929141 2023-03-24 01:55:29,668 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/9a9d07912dd049a89c384fb9d56f3cbf as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9a9d07912dd049a89c384fb9d56f3cbf 2023-03-24 01:55:29,673 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:29,689 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into 9a9d07912dd049a89c384fb9d56f3cbf(size=25.6 K), total size for store is 83.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:29,689 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,689 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=13, startTime=1679622928803; duration=0sec 2023-03-24 01:55:29,689 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:29,689 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:29,691 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:29,691 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:29,691 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,691 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/354fd785fd9942d6b25d295ed79ba45f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=35.6 K 2023-03-24 01:55:29,691 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 354fd785fd9942d6b25d295ed79ba45f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:29,692 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting a115d4086a3547b9a5aa97817a177006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679622927940 2023-03-24 01:55:29,692 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e2d5baff3dc14714b399a230a62524b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679622929141 2023-03-24 01:55:29,692 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:29,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:29,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:29,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:29,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:29,698 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:29,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:29,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:29,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,699 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,700 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#348 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:29,703 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:29,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:29,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:29,741 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,741 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:29,741 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:29,741 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:29,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,746 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:29,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51df5acc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:29,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:29,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,751 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:29,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:29,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:29,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,756 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/77c22e51acea433ba2a4a7898423be1d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/77c22e51acea433ba2a4a7898423be1d 2023-03-24 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:29,757 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:29,757 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:29,757 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:29,758 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:29,758 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:29,759 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,759 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,759 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,761 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into 77c22e51acea433ba2a4a7898423be1d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:29,761 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:29,761 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=13, startTime=1679622929474; duration=0sec 2023-03-24 01:55:29,761 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622989747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622989758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622989759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622989761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:29,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:29,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,770 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,770 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:29,770 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:29,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622989762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622989863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,870 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:29,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:29,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:29,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:29,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:29,873 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:29,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:29,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:29,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,875 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:29,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,876 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:29,877 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,877 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:29,877 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:29,877 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:29,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:29,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,879 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:29,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:29,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abb637d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:29,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:29,881 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:29,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:29,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:29,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622989875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:29,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:29,886 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:29,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:29,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:29,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:29,886 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:29,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:29,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:29,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:29,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,890 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:29,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:29,893 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,893 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,893 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:29,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:29,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:29,894 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:29,894 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:29,994 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:29,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:29,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:29,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:29,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:29,997 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:29,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:29,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:29,999 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:29,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:29,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:30,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:30,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:30,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:30,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:30,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:30,002 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:30,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:30,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:30,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:30,006 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:30,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37d6251d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:30,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:30,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:30,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:30,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:30,013 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/c9e3bd4a7a6b4464a454f2a61a2b5618 2023-03-24 01:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:30,017 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:30,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:30,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:30,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:30,017 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:30,019 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,019 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,019 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,025 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,025 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:30,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:30,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:30,025 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,025 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:30,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622990071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622990084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,109 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/872b0e1769c34a69b35d149a2c3d25b6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/872b0e1769c34a69b35d149a2c3d25b6 2023-03-24 01:55:30,112 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into 872b0e1769c34a69b35d149a2c3d25b6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:30,113 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:30,113 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=13, startTime=1679622929474; duration=0sec 2023-03-24 01:55:30,113 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:30,125 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:30,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:30,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:30,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:30,128 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:30,128 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,128 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:30,128 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:30,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,130 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:30,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:30,131 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,131 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:30,132 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:30,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:30,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,136 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee3e2dd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:30,137 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:30,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,140 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:30,140 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:30,140 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:30,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,140 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:30,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,142 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:30,145 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:30,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:30,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:30,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:30,145 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:30,146 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,147 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,147 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,150 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:30,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:30,152 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,153 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:30,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,153 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:30,153 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,153 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,253 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:30,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:30,256 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:30,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:30,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:30,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,258 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,259 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,259 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,259 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:30,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:30,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:30,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:30,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:30,262 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,263 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,263 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:30,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4541dcf7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:30,266 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:30,266 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,267 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:30,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:30,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:30,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,268 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:30,269 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,269 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622990265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622990267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622990267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:30,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:30,272 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:30,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:30,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:30,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:30,272 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:30,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,276 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:30,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:30,280 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,280 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,280 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:30,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:30,280 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,281 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,381 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:30,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:30,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:30,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:30,384 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:30,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:30,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:30,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622990380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,386 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:30,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:30,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:30,387 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:30,389 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:30,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5470efa9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:30,391 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:30,391 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622990388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,392 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:30,393 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:30,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:30,394 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:30,398 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:30,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:30,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:30,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:30,398 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:30,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,400 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,400 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:30,400 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,402 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:30,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:30,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:30,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:30,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:30,404 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:30,404 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:30,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:30,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d6706695bb9e4174b9741b2b9360b691 2023-03-24 01:55:30,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/6d42d813a581471eb8ac24e8f6fd5844 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844 2023-03-24 01:55:30,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844, entries=250, sequenceid=201, filesize=38.9 K 2023-03-24 01:55:30,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/c9e3bd4a7a6b4464a454f2a61a2b5618 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618 2023-03-24 01:55:30,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618, entries=150, sequenceid=201, filesize=11.8 K 2023-03-24 01:55:30,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d6706695bb9e4174b9741b2b9360b691 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691 2023-03-24 01:55:30,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691, entries=150, sequenceid=201, filesize=11.8 K 2023-03-24 01:55:30,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a5a2c70f4dec9fd5af77e99de4acd24f in 897ms, sequenceid=201, compaction requested=true 2023-03-24 01:55:30,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:30,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:30,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:30,446 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:30,446 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:55:30,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:30,447 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:55:30,447 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:55:30,447 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. because compaction request was cancelled 2023-03-24 01:55:30,447 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:30,448 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,448 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9a9d07912dd049a89c384fb9d56f3cbf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=122.0 K 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. because compaction request was cancelled 2023-03-24 01:55:30,448 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9a9d07912dd049a89c384fb9d56f3cbf, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679622927611 2023-03-24 01:55:30,449 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 21f5f964f8064af7b37c7797f984dd4b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1679622927940 2023-03-24 01:55:30,449 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 280258a2e3834518ae0340491a2facfd, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679622929133 2023-03-24 01:55:30,449 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6d42d813a581471eb8ac24e8f6fd5844, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679622929430 2023-03-24 01:55:30,457 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#350 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:30,504 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:30,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:30,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:30,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:30,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:30,507 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:30,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:30,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:30,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,510 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:30,510 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:30,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:30,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:30,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:30,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:30,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:30,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:30,512 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:30,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:30,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:30,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622990912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:30,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56622 deadline: 1679622990912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:30,935 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f926bb5e8b3f4448823a26ba7e46edda to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f926bb5e8b3f4448823a26ba7e46edda 2023-03-24 01:55:30,940 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/e8346a05f44f4e0e8bf74ece1e555614 2023-03-24 01:55:31,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622991017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622991023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1679622991225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622991229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,275 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/4b6064f8723a4917b4c2ada908e15fe8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4b6064f8723a4917b4c2ada908e15fe8 2023-03-24 01:55:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622991271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622991273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,279 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into 4b6064f8723a4917b4c2ada908e15fe8(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:31,279 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:31,279 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=12, startTime=1679622930446; duration=0sec 2023-03-24 01:55:31,279 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:31,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622991279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,351 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/922db6903c054d92a0db20c15ffd2817 2023-03-24 01:55:31,360 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/f211bef68e7140169e610fd6a8c7b50b 2023-03-24 01:55:31,363 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/e8346a05f44f4e0e8bf74ece1e555614 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614 2023-03-24 01:55:31,366 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614, entries=150, sequenceid=241, filesize=25.3 K 2023-03-24 01:55:31,367 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/922db6903c054d92a0db20c15ffd2817 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817 2023-03-24 01:55:31,370 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817, entries=150, sequenceid=241, filesize=11.8 K 2023-03-24 01:55:31,370 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/f211bef68e7140169e610fd6a8c7b50b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b 2023-03-24 01:55:31,374 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b, entries=150, sequenceid=241, filesize=11.8 K 2023-03-24 01:55:31,374 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a5a2c70f4dec9fd5af77e99de4acd24f in 862ms, sequenceid=241, compaction requested=true 2023-03-24 01:55:31,374 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:31,375 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:31,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:31,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:31,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:31,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:31,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5208d6f7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:31,378 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:31,378 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:31,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:31,379 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:31,389 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:31,392 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:31,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:31,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:31,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:31,392 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:31,393 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,393 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:31,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,393 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:31,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,394 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:31,395 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:31,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:31,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:31,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:31,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:31,398 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:31,398 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:31,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:31,498 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:31,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:31,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:31,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:31,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:31,501 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:31,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:31,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:31,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:31,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:31,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:31,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:31,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:31,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:31,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:31,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:31,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:31,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:31,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:31,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:31,505 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:31,505 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:31,516 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c053550f9e77482e964dc34aab37614a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c053550f9e77482e964dc34aab37614a 2023-03-24 01:55:31,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:31,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:31,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622991581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622931568, max timestamp=1679622931583Num uniques 859; 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622931568, max timestamp=1679622931583Num uniques 859; 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622931568, max timestamp=1679622931583Num uniques 859; 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:31,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:31,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622991669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622991687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622991779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622991898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:31,921 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=253, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/55564e13566b4a4a9e5a28183e3c597a 2023-03-24 01:55:31,930 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/222d7290d7ca4f678fe258e34a231b25 2023-03-24 01:55:32,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622991996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622992208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622992303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,343 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/17f8991a220642f6afe1f4b2f98f8664 2023-03-24 01:55:32,348 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/55564e13566b4a4a9e5a28183e3c597a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a 2023-03-24 01:55:32,352 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a, entries=150, sequenceid=253, filesize=25.3 K 2023-03-24 01:55:32,353 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/222d7290d7ca4f678fe258e34a231b25 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25 2023-03-24 01:55:32,356 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25, entries=150, sequenceid=253, filesize=11.8 K 2023-03-24 01:55:32,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/17f8991a220642f6afe1f4b2f98f8664 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664 2023-03-24 01:55:32,360 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664, entries=150, sequenceid=253, filesize=11.8 K 2023-03-24 01:55:32,360 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a5a2c70f4dec9fd5af77e99de4acd24f in 855ms, sequenceid=253, compaction requested=true 2023-03-24 01:55:32,360 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:32,360 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:32,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:32,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:32,366 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,366 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,367 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,367 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:32,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@238c3361[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:32,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:32,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:32,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:32,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:32,371 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:32,374 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:32,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:32,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:32,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:32,374 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:32,375 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,375 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,375 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:32,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,377 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:32,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:32,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:32,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:32,380 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:32,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:32,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:32,381 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:32,381 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:32,481 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:32,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:32,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:32,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:32,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:32,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:32,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:32,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:32,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:32,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:32,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:32,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:32,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:32,487 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:32,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:32,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:32,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:32,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:32,488 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:32,488 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:32,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:32,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:32,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622992774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622992809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:32,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622992883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,902 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032400df78b9e91d44a190960fc4a690b2b0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400df78b9e91d44a190960fc4a690b2b0 2023-03-24 01:55:32,909 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=288, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/8cf27783d7524a159c5d15a88ab9dbda 2023-03-24 01:55:32,920 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4c58d88217474b26bff668857b97f230 2023-03-24 01:55:32,929 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/106663453846401dad0433adcc5a96c3 2023-03-24 01:55:32,933 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/8cf27783d7524a159c5d15a88ab9dbda as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda 2023-03-24 01:55:32,937 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda, entries=150, sequenceid=288, filesize=25.4 K 2023-03-24 01:55:32,937 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4c58d88217474b26bff668857b97f230 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230 2023-03-24 01:55:32,941 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230, entries=150, sequenceid=288, filesize=11.9 K 2023-03-24 01:55:32,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/106663453846401dad0433adcc5a96c3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3 2023-03-24 01:55:32,946 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3, entries=150, sequenceid=288, filesize=11.9 K 2023-03-24 01:55:32,946 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a5a2c70f4dec9fd5af77e99de4acd24f in 458ms, sequenceid=288, compaction requested=true 2023-03-24 01:55:32,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:32,947 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:32,950 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,951 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:32,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e1e583[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:32,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:32,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:32,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:32,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:32,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:32,958 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:32,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:32,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:32,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:32,958 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:32,960 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:32,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:32,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,962 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:32,967 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:32,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:32,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:32,967 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:32,967 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:32,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:32,968 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:32,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:32,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:33,067 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:33,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:33,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:33,072 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:33,072 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:33,072 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:33,072 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:33,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:33,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:33,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:33,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:33,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:33,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:33,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:33,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:33,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:33,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:33,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:33,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:33,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:33,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:33,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:33,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:33,076 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:33,076 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:33,076 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:33,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:33,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:33,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622933165, max timestamp=1679622933201Num uniques 859; 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622933165, max timestamp=1679622933201Num uniques 859; 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622933165, max timestamp=1679622933201Num uniques 859; 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:33,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:33,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622993304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622993304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622993304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,315 DEBUG [Thread-2326] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:55:33,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622993311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622993414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622993414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622993418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,496 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032416ab429191b6418eaef7ccf35e8be96e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032416ab429191b6418eaef7ccf35e8be96e 2023-03-24 01:55:33,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622993620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622993620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622993621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622993823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,901 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/342d09bfaeac4ab4936f483a7a58e235 2023-03-24 01:55:33,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622993926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622993926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:33,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:33,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622993928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,310 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/2da7495f0cac4ba98cf7a7f1bff495c3 2023-03-24 01:55:34,324 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ac37f06b85b843be9407d139b40600ab 2023-03-24 01:55:34,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/342d09bfaeac4ab4936f483a7a58e235 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235 2023-03-24 01:55:34,332 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235, entries=150, sequenceid=299, filesize=25.4 K 2023-03-24 01:55:34,332 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/2da7495f0cac4ba98cf7a7f1bff495c3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3 2023-03-24 01:55:34,336 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3, entries=150, sequenceid=299, filesize=11.9 K 2023-03-24 01:55:34,337 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ac37f06b85b843be9407d139b40600ab as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab 2023-03-24 01:55:34,341 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab, entries=150, sequenceid=299, filesize=11.9 K 2023-03-24 01:55:34,342 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a5a2c70f4dec9fd5af77e99de4acd24f in 1266ms, sequenceid=299, compaction requested=true 2023-03-24 01:55:34,342 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:34,342 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3f4291[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,352 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,352 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,365 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,366 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,366 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,365 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:34,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 01:55:34,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622933263, max timestamp=1679622934438Num uniques 1018; 2023-03-24 01:55:34,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622933263, max timestamp=1679622934438Num uniques 1018; 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:34,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:34,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:34,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 01:55:34,466 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:34,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:34,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:34,469 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:34,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:34,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:34,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:34,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:34,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,475 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f14846f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,484 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,488 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,488 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622994491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622994492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622994498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,588 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:34,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:34,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:34,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:34,591 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:34,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:34,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:34,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:34,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc1add7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,603 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,607 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622994601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622994601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:34,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,611 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,611 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622994606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,711 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:34,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:34,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:34,714 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:34,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:34,714 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:34,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,715 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,717 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:34,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:34,719 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,719 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:34,719 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:34,719 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6b8011[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,723 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,723 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,723 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,727 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,727 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:34,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,741 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,741 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622994809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622994810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:34,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622994815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,841 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:34,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:34,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:34,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:34,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:34,845 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:34,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:34,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:34,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:34,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:34,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:34,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,848 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:34,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:34,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,852 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,852 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e3d8390d460e4a7995d32e411a704daf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3d8390d460e4a7995d32e411a704daf 2023-03-24 01:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b017007[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,860 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,860 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,860 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,860 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,864 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,864 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:34,870 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,870 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,970 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:34,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:34,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:34,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:34,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:34,974 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:34,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:34,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:34,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,975 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,975 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:34,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:34,977 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,977 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:34,977 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:34,977 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:34,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,979 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40bda2f4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:34,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:34,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,981 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:34,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:34,984 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:34,987 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:34,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:34,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:34,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:34,987 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:34,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:34,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:34,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:34,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:34,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:34,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:34,994 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:34,994 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:34,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:34,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:34,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,095 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,098 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,100 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,101 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,101 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,101 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,101 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,104 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f21021[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,107 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,107 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,108 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,110 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,111 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,120 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,120 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,120 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56648 deadline: 1679622995120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622995122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622995125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,220 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,223 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,224 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,225 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,226 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,227 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,227 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,227 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,227 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@764f6726[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,230 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,230 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,233 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,235 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,236 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,243 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,244 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,244 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,257 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/86133a44b24644e898d642239c3c0fae 2023-03-24 01:55:35,344 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,347 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,351 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360ea0a0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,356 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,360 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,360 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,367 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,367 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,467 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,470 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,471 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,474 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,475 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,475 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b091e97[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,479 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,489 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,489 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,489 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,589 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,593 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,597 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b4a76cf[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,616 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,616 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,616 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622995631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56680 deadline: 1679622995631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622995636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/98e1300d6b814d75988b481abcf23131 2023-03-24 01:55:35,717 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,720 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,721 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,723 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,725 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263152d7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,727 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,729 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,729 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,729 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,731 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,733 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,733 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,739 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,739 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,839 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:35,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622995833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,845 DEBUG [Thread-2328] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:55:35,845 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,847 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f9fd11[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,853 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,854 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,854 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,858 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,858 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,858 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,858 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,858 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,860 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,860 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,860 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,865 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,865 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,965 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:35,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:35,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:35,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:35,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:35,968 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:35,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:35,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:35,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:35,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:35,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:35,972 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:35,973 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,973 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:35,973 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:35,973 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:35,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:35,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,974 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:35,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:35,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3657ba9f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:35,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:35,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:35,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:35,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:35,982 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:35,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:35,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:35,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:35,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:35,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:35,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:35,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:35,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:35,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:35,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:35,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:35,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:35,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:35,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:35,991 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:35,991 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:35,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:35,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/babae0cc05cc4d6e864925c080e6c952 2023-03-24 01:55:36,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/86133a44b24644e898d642239c3c0fae as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae 2023-03-24 01:55:36,091 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:36,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:36,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae, entries=250, sequenceid=337, filesize=39.2 K 2023-03-24 01:55:36,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/98e1300d6b814d75988b481abcf23131 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131 2023-03-24 01:55:36,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:36,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:36,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:36,095 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:36,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:36,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:36,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131, entries=150, sequenceid=337, filesize=11.9 K 2023-03-24 01:55:36,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/babae0cc05cc4d6e864925c080e6c952 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952 2023-03-24 01:55:36,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:36,100 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952, entries=150, sequenceid=337, filesize=11.9 K 2023-03-24 01:55:36,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:36,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~188.91 KB/193440, currentSize=114.05 KB/116790 for a5a2c70f4dec9fd5af77e99de4acd24f in 1663ms, sequenceid=337, compaction requested=true 2023-03-24 01:55:36,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:36,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:36,101 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:55:36,101 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 01:55:36,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:36,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:36,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:36,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:36,102 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:36,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,103 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170388 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:55:36,103 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:36,104 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,104 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4b6064f8723a4917b4c2ada908e15fe8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=166.4 K 2023-03-24 01:55:36,104 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85146 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 01:55:36,104 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:36,104 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,104 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/872b0e1769c34a69b35d149a2c3d25b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=83.2 K 2023-03-24 01:55:36,104 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4b6064f8723a4917b4c2ada908e15fe8, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679622929430 2023-03-24 01:55:36,105 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e8346a05f44f4e0e8bf74ece1e555614, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1679622929741 2023-03-24 01:55:36,106 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 55564e13566b4a4a9e5a28183e3c597a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679622930897 2023-03-24 01:55:36,106 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 872b0e1769c34a69b35d149a2c3d25b6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679622929141 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8cf27783d7524a159c5d15a88ab9dbda, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679622931626 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c9e3bd4a7a6b4464a454f2a61a2b5618, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679622929430 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 342d09bfaeac4ab4936f483a7a58e235, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622932725 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 922db6903c054d92a0db20c15ffd2817, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1679622929741 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 86133a44b24644e898d642239c3c0fae, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933263 2023-03-24 01:55:36,107 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 222d7290d7ca4f678fe258e34a231b25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679622930897 2023-03-24 01:55:36,108 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4c58d88217474b26bff668857b97f230, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679622931626 2023-03-24 01:55:36,108 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2da7495f0cac4ba98cf7a7f1bff495c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622932725 2023-03-24 01:55:36,109 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 98e1300d6b814d75988b481abcf23131, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933281 2023-03-24 01:55:36,127 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#368 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:36,128 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032423fa023f25b8448d83a1f1e1ffb10163 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032423fa023f25b8448d83a1f1e1ffb10163 2023-03-24 01:55:36,139 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#367 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:36,143 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/afe14470a545442bb28088080dc59f6d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/afe14470a545442bb28088080dc59f6d 2023-03-24 01:55:36,148 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into afe14470a545442bb28088080dc59f6d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:36,148 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,148 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=9, startTime=1679622936101; duration=0sec 2023-03-24 01:55:36,148 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:36,148 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 01:55:36,149 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85146 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 01:55:36,149 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:36,150 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,150 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/77c22e51acea433ba2a4a7898423be1d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=83.2 K 2023-03-24 01:55:36,150 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 77c22e51acea433ba2a4a7898423be1d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679622929141 2023-03-24 01:55:36,150 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting d6706695bb9e4174b9741b2b9360b691, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679622929430 2023-03-24 01:55:36,150 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f211bef68e7140169e610fd6a8c7b50b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1679622929741 2023-03-24 01:55:36,151 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 17f8991a220642f6afe1f4b2f98f8664, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679622930897 2023-03-24 01:55:36,151 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 106663453846401dad0433adcc5a96c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1679622931626 2023-03-24 01:55:36,151 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ac37f06b85b843be9407d139b40600ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1679622932725 2023-03-24 01:55:36,151 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting babae0cc05cc4d6e864925c080e6c952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933281 2023-03-24 01:55:36,152 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/324396a349b845fea05fab8ee552e8fd 2023-03-24 01:55:36,170 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#370 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:36,177 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/67a12fc2529b45afbfe8dd9d6ef0c3df 2023-03-24 01:55:36,183 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/e12ec6f3fc9c423aa758603f0e096ed4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e12ec6f3fc9c423aa758603f0e096ed4 2023-03-24 01:55:36,188 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/2a3dd718d0e5482692708b79edc95f48 2023-03-24 01:55:36,189 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into e12ec6f3fc9c423aa758603f0e096ed4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:36,189 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,189 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=9, startTime=1679622936101; duration=0sec 2023-03-24 01:55:36,189 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:36,193 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/324396a349b845fea05fab8ee552e8fd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd 2023-03-24 01:55:36,198 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd, entries=150, sequenceid=357, filesize=25.4 K 2023-03-24 01:55:36,198 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/67a12fc2529b45afbfe8dd9d6ef0c3df as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df 2023-03-24 01:55:36,202 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df, entries=150, sequenceid=357, filesize=11.9 K 2023-03-24 01:55:36,202 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/2a3dd718d0e5482692708b79edc95f48 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48 2023-03-24 01:55:36,209 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48, entries=150, sequenceid=357, filesize=11.9 K 2023-03-24 01:55:36,210 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for a5a2c70f4dec9fd5af77e99de4acd24f in 108ms, sequenceid=357, compaction requested=false 2023-03-24 01:55:36,210 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,210 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:36,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed35e94[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:36,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:36,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:36,215 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,215 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:36,217 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:36,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:36,220 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:36,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:36,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:36,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:36,220 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:36,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,225 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:36,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:36,229 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:36,229 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:36,329 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:36,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:36,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:36,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:36,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:36,333 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:36,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:36,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:36,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,335 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:36,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:36,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:36,337 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,337 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:36,337 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,337 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:36,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,339 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:36,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62d80a73[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:36,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:36,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,343 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:36,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:36,345 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:36,348 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:36,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:36,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:36,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:36,348 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:36,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,350 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:36,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:36,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:36,356 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:36,356 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:36,456 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:36,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:36,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:36,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:36,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:36,460 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:36,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:36,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:36,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:36,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:36,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:36,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:36,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:36,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:36,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:36,465 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,465 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:36,466 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,466 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:36,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:36,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:36,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e3624c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:36,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:36,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:36,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:36,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:36,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:36,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:36,476 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:36,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:36,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:36,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:36,476 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:36,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,481 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:36,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:36,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,486 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:36,486 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:36,485 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,586 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:36,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:36,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:36,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:36,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:36,592 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:36,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:36,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:36,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,596 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:36,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:36,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:36,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,603 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/63bfeb65ef014cb4962f563a3bd31865 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/63bfeb65ef014cb4962f563a3bd31865 2023-03-24 01:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436baa8d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:36,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:36,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:36,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:36,607 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into 63bfeb65ef014cb4962f563a3bd31865(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:36,607 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,607 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=10, startTime=1679622936101; duration=0sec 2023-03-24 01:55:36,607 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,607 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:36,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:36,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:36,610 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:36,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:36,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:36,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:36,610 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:36,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,614 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:36,623 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,623 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:36,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:36,624 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,624 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,624 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,624 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:36,624 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:36,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,724 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:36,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:36,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:36,732 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:36,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:36,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:36,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:36,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:36,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:36,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:36,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:36,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:36,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:36,764 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,764 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:36,764 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 01:55:36,768 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:36,769 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,769 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:36,769 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,769 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:36,769 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:36,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:36,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:36,784 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324dbaa04b355654b2c82c7f4ee2d0b4775 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dbaa04b355654b2c82c7f4ee2d0b4775 2023-03-24 01:55:36,793 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/302a74a0d041475db31bb67cc7bb6af7 2023-03-24 01:55:36,814 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/ae85379bb419458d99588cb39f9094e4 2023-03-24 01:55:36,874 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d3c28cc18130447097ac18a7a72462e9 2023-03-24 01:55:36,878 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/302a74a0d041475db31bb67cc7bb6af7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7 2023-03-24 01:55:36,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622936853, max timestamp=1679622936878Num uniques 859; 2023-03-24 01:55:36,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:36,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622936853, max timestamp=1679622936878Num uniques 859; 2023-03-24 01:55:36,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622936853, max timestamp=1679622936878Num uniques 859; 2023-03-24 01:55:36,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:55:36,884 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7, entries=200, sequenceid=372, filesize=32.3 K 2023-03-24 01:55:36,885 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/ae85379bb419458d99588cb39f9094e4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4 2023-03-24 01:55:36,890 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4, entries=150, sequenceid=372, filesize=11.9 K 2023-03-24 01:55:36,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/d3c28cc18130447097ac18a7a72462e9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9 2023-03-24 01:55:36,895 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9, entries=100, sequenceid=372, filesize=9.5 K 2023-03-24 01:55:36,896 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=60.38 KB/61830 for a5a2c70f4dec9fd5af77e99de4acd24f in 132ms, sequenceid=372, compaction requested=true 2023-03-24 01:55:36,896 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:36,896 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:36,896 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:36,896 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:36,896 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,896 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:36,896 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:36,898 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:36,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=73.80 KB heapSize=201 KB 2023-03-24 01:55:36,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:36,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17187b8a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:36,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:36,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:36,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:36,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:36,904 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:36,904 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:36,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:36,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,907 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,907 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:36,910 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:36,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:36,911 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:36,911 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:36,911 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:36,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,916 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,916 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,916 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:36,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:36,919 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:36,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:36,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:36,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,919 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:36,919 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:36,919 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:36,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:36,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:36,962 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ce1056f9af224a7f831662b7d84df85f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ce1056f9af224a7f831662b7d84df85f 2023-03-24 01:55:36,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:36,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622996967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:36,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622996968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:36,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:36,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622996968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,020 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/e77e721c34584cb2bc7376476309c9cd 2023-03-24 01:55:37,020 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:37,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:37,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:37,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:37,036 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:37,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:37,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:37,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:37,037 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,037 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:37,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:37,041 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,042 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,042 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:37,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:37,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:37,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:37,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:37,072 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:37,076 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:37,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:37,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35316088[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:37,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:37,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:37,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:37,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:37,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:37,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:37,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:37,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:37,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:37,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/83c218e8924a4185aede095f03b0cb0b 2023-03-24 01:55:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622997094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622997094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:37,104 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:37,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:37,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:37,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:37,104 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:37,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622997097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:37,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:37,108 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:37,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:37,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:37,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:37,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:37,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:37,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:37,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:37,118 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:37,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,118 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:37,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/58e731e88ec3453ea09c252e01c86649 2023-03-24 01:55:37,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/e77e721c34584cb2bc7376476309c9cd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd 2023-03-24 01:55:37,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd, entries=250, sequenceid=400, filesize=39.2 K 2023-03-24 01:55:37,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/83c218e8924a4185aede095f03b0cb0b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b 2023-03-24 01:55:37,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b, entries=150, sequenceid=400, filesize=11.9 K 2023-03-24 01:55:37,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/58e731e88ec3453ea09c252e01c86649 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649 2023-03-24 01:55:37,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649, entries=150, sequenceid=400, filesize=11.9 K 2023-03-24 01:55:37,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~220.88 KB/226176, currentSize=107.34 KB/109920 for a5a2c70f4dec9fd5af77e99de4acd24f in 268ms, sequenceid=400, compaction requested=true 2023-03-24 01:55:37,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:37,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:37,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:37,169 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:37,169 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:37,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:37,171 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49422 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:37,171 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/B is initiating minor compaction (all files) 2023-03-24 01:55:37,171 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/B in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,171 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/afe14470a545442bb28088080dc59f6d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=48.3 K 2023-03-24 01:55:37,172 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting afe14470a545442bb28088080dc59f6d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933281 2023-03-24 01:55:37,173 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 67a12fc2529b45afbfe8dd9d6ef0c3df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622934487 2023-03-24 01:55:37,173 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ae85379bb419458d99588cb39f9094e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679622936683 2023-03-24 01:55:37,173 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 83c218e8924a4185aede095f03b0cb0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622936895 2023-03-24 01:55:37,174 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125986 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:37,174 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/A is initiating minor compaction (all files) 2023-03-24 01:55:37,174 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/A in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,174 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/63bfeb65ef014cb4962f563a3bd31865, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=123.0 K 2023-03-24 01:55:37,175 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 63bfeb65ef014cb4962f563a3bd31865, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933281 2023-03-24 01:55:37,175 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 324396a349b845fea05fab8ee552e8fd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622934487 2023-03-24 01:55:37,176 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 302a74a0d041475db31bb67cc7bb6af7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679622936683 2023-03-24 01:55:37,176 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e77e721c34584cb2bc7376476309c9cd, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622936869 2023-03-24 01:55:37,198 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#B#compaction#379 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:37,218 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:37,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:37,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:37,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:37,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:37,228 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:37,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:37,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:37,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:37,230 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#A#compaction#378 average throughput is 0.45 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:37,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:37,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:37,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,233 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:37,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:37,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:37,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:37,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:37,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:37,244 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,244 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:37,245 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:37,261 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/f91c8901b3c94191bf09cc48b6fc714c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/f91c8901b3c94191bf09cc48b6fc714c 2023-03-24 01:55:37,269 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/B of a5a2c70f4dec9fd5af77e99de4acd24f into f91c8901b3c94191bf09cc48b6fc714c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:37,269 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:37,269 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/B, priority=12, startTime=1679622937169; duration=0sec 2023-03-24 01:55:37,270 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:37,270 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:55:37,273 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:55:37,273 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): a5a2c70f4dec9fd5af77e99de4acd24f/C is initiating minor compaction (all files) 2023-03-24 01:55:37,273 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5a2c70f4dec9fd5af77e99de4acd24f/C in TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:37,274 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e12ec6f3fc9c423aa758603f0e096ed4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp, totalSize=45.9 K 2023-03-24 01:55:37,274 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e12ec6f3fc9c423aa758603f0e096ed4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679622933281 2023-03-24 01:55:37,274 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 2a3dd718d0e5482692708b79edc95f48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622934487 2023-03-24 01:55:37,275 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d3c28cc18130447097ac18a7a72462e9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1679622936683 2023-03-24 01:55:37,276 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 58e731e88ec3453ea09c252e01c86649, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1679622936895 2023-03-24 01:55:37,305 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5a2c70f4dec9fd5af77e99de4acd24f#C#compaction#381 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:37,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:37,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:37,342 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/f8c39badccde49eda7f6d9729d820774 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f8c39badccde49eda7f6d9729d820774 2023-03-24 01:55:37,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622997339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622997340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56618 deadline: 1679622997342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,350 DEBUG [Thread-2326] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:55:37,351 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/C of a5a2c70f4dec9fd5af77e99de4acd24f into f8c39badccde49eda7f6d9729d820774(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:37,351 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:37,351 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/C, priority=12, startTime=1679622937169; duration=0sec 2023-03-24 01:55:37,351 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:37,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622997345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622997447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622997449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622997454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622997655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622997655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622997661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,702 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e82bbcf1901d496ab7da39db6e0160ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e82bbcf1901d496ab7da39db6e0160ab 2023-03-24 01:55:37,719 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/a8990f3513ff44dcad8e7957f2ac79b4 2023-03-24 01:55:37,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622997962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622997964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:37,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:37,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622997968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,128 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/503e22bea8334f3791b402ba05bb151e 2023-03-24 01:55:38,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622998467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:38,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622998470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:38,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622998474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,511 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/a053b08a36e844028008bfbd38c324e6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a053b08a36e844028008bfbd38c324e6 2023-03-24 01:55:38,515 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5a2c70f4dec9fd5af77e99de4acd24f/A of a5a2c70f4dec9fd5af77e99de4acd24f into a053b08a36e844028008bfbd38c324e6(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 01:55:38,515 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:38,515 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., storeName=a5a2c70f4dec9fd5af77e99de4acd24f/A, priority=12, startTime=1679622937169; duration=1sec 2023-03-24 01:55:38,515 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:38,541 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/3ee4614becb94460921430f603395039 2023-03-24 01:55:38,544 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/a8990f3513ff44dcad8e7957f2ac79b4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a8990f3513ff44dcad8e7957f2ac79b4 2023-03-24 01:55:38,547 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a8990f3513ff44dcad8e7957f2ac79b4, entries=150, sequenceid=419, filesize=25.4 K 2023-03-24 01:55:38,548 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/503e22bea8334f3791b402ba05bb151e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/503e22bea8334f3791b402ba05bb151e 2023-03-24 01:55:38,558 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/503e22bea8334f3791b402ba05bb151e, entries=150, sequenceid=419, filesize=11.9 K 2023-03-24 01:55:38,559 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/3ee4614becb94460921430f603395039 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/3ee4614becb94460921430f603395039 2023-03-24 01:55:38,562 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/3ee4614becb94460921430f603395039, entries=150, sequenceid=419, filesize=11.9 K 2023-03-24 01:55:38,563 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a5a2c70f4dec9fd5af77e99de4acd24f in 1319ms, sequenceid=419, compaction requested=false 2023-03-24 01:55:38,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:38,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:38,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:38,565 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:38,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:38,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:38,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66938560[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:38,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:38,567 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:38,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:38,572 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:38,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:38,574 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:38,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:38,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:38,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:38,574 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:38,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:38,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,579 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:38,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:38,584 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:38,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:38,584 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:38,584 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:38,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:38,684 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:38,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:38,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:38,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:38,688 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:38,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:38,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:38,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:38,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:38,689 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:38,689 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:38,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:38,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:38,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:38,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:38,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:38,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:38,691 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:38,691 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:38,691 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:38,692 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:39,104 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032490307f9fb8254b9ea45bb761a335c5d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032490307f9fb8254b9ea45bb761a335c5d3 2023-03-24 01:55:39,109 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=435, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/48cb5aa6bdf14747bb6d98c471941f69 2023-03-24 01:55:39,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:39,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:39,526 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4236014db93d44d38d9f678f5ed49d79 2023-03-24 01:55:39,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622999516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622999533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622999533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,553 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 01:55:39,631 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/43312bcd6f2b42bb90a631de192c7938, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4f88b93142824e849a09498c48bd0d48, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9a9d07912dd049a89c384fb9d56f3cbf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4b6064f8723a4917b4c2ada908e15fe8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/63bfeb65ef014cb4962f563a3bd31865, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd] to archive 2023-03-24 01:55:39,632 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:39,635 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9ec0c5a0bda5463f9f63933eba2ada77 2023-03-24 01:55:39,636 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/68518042d6be4807bba7937abfaefb73 2023-03-24 01:55:39,638 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/ef2277e0178f45c78fded6a8263fecd1 2023-03-24 01:55:39,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,639 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/43312bcd6f2b42bb90a631de192c7938 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/43312bcd6f2b42bb90a631de192c7938 2023-03-24 01:55:39,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622999634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,641 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/f3c967c49abf46b8a39847f2b47675b6 2023-03-24 01:55:39,642 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/b2c51f50230c43378faea56e0edcf822 2023-03-24 01:55:39,643 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/c102b9ef1dfc4272b79f2f1d0abddca2 2023-03-24 01:55:39,646 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4f88b93142824e849a09498c48bd0d48 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4f88b93142824e849a09498c48bd0d48 2023-03-24 01:55:39,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622999642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622999642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,648 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/330991ff60714c0e875bea7f6e5bd365 2023-03-24 01:55:39,649 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/7c4cb8ddeb76480294c79d49f79b1351 2023-03-24 01:55:39,650 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9a9d07912dd049a89c384fb9d56f3cbf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9a9d07912dd049a89c384fb9d56f3cbf 2023-03-24 01:55:39,651 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/21f5f964f8064af7b37c7797f984dd4b 2023-03-24 01:55:39,652 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/280258a2e3834518ae0340491a2facfd 2023-03-24 01:55:39,653 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/6d42d813a581471eb8ac24e8f6fd5844 2023-03-24 01:55:39,656 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4b6064f8723a4917b4c2ada908e15fe8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/4b6064f8723a4917b4c2ada908e15fe8 2023-03-24 01:55:39,659 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e8346a05f44f4e0e8bf74ece1e555614 2023-03-24 01:55:39,665 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/55564e13566b4a4a9e5a28183e3c597a 2023-03-24 01:55:39,672 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/8cf27783d7524a159c5d15a88ab9dbda 2023-03-24 01:55:39,673 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/342d09bfaeac4ab4936f483a7a58e235 2023-03-24 01:55:39,676 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/86133a44b24644e898d642239c3c0fae 2023-03-24 01:55:39,678 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/63bfeb65ef014cb4962f563a3bd31865 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/63bfeb65ef014cb4962f563a3bd31865 2023-03-24 01:55:39,679 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/324396a349b845fea05fab8ee552e8fd 2023-03-24 01:55:39,680 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/302a74a0d041475db31bb67cc7bb6af7 2023-03-24 01:55:39,682 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/e77e721c34584cb2bc7376476309c9cd 2023-03-24 01:55:39,688 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b02f4061280143cfbb68aed6079a2ac0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/701edbfedef04eb8a087f99358e929e8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/3b186e7595d6400aa7d352b02e296885, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/872b0e1769c34a69b35d149a2c3d25b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/afe14470a545442bb28088080dc59f6d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b] to archive 2023-03-24 01:55:39,689 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:39,690 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9b0e4e933b52412d8023dbcbb2059816 2023-03-24 01:55:39,691 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/9393885984974927bef8353ab580b72a 2023-03-24 01:55:39,693 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b02f4061280143cfbb68aed6079a2ac0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b02f4061280143cfbb68aed6079a2ac0 2023-03-24 01:55:39,695 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/7d00a1813ce14affbd2bf7f45c149e27 2023-03-24 01:55:39,697 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/524b8594a8b84e4b87dcc049ee37e2d4 2023-03-24 01:55:39,698 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2a8742d03084452bb3922dd5d88b7a34 2023-03-24 01:55:39,698 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/701edbfedef04eb8a087f99358e929e8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/701edbfedef04eb8a087f99358e929e8 2023-03-24 01:55:39,699 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/99c75f6bb74a46e19cf8b6e301be359e 2023-03-24 01:55:39,701 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c8ad3316900c4ef5bb720382d0c914aa 2023-03-24 01:55:39,702 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/3b186e7595d6400aa7d352b02e296885 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/3b186e7595d6400aa7d352b02e296885 2023-03-24 01:55:39,703 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/d004583d9afe4f5ea01e1aaed454ef1d 2023-03-24 01:55:39,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/b87b7a4957534b0ca96451b9e34d8c7a 2023-03-24 01:55:39,707 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/872b0e1769c34a69b35d149a2c3d25b6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/872b0e1769c34a69b35d149a2c3d25b6 2023-03-24 01:55:39,708 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/db2dcdd7171547679c4d8c471d5a1f02 2023-03-24 01:55:39,709 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/c9e3bd4a7a6b4464a454f2a61a2b5618 2023-03-24 01:55:39,710 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/922db6903c054d92a0db20c15ffd2817 2023-03-24 01:55:39,712 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/222d7290d7ca4f678fe258e34a231b25 2023-03-24 01:55:39,713 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4c58d88217474b26bff668857b97f230 2023-03-24 01:55:39,713 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/2da7495f0cac4ba98cf7a7f1bff495c3 2023-03-24 01:55:39,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/afe14470a545442bb28088080dc59f6d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/afe14470a545442bb28088080dc59f6d 2023-03-24 01:55:39,716 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/98e1300d6b814d75988b481abcf23131 2023-03-24 01:55:39,717 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/67a12fc2529b45afbfe8dd9d6ef0c3df 2023-03-24 01:55:39,718 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/ae85379bb419458d99588cb39f9094e4 2023-03-24 01:55:39,719 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/83c218e8924a4185aede095f03b0cb0b 2023-03-24 01:55:39,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/fdf17f4593954afdb5fc8f7092184df7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cc97830cd3ea4aeaa1a3a9180b444605, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/354fd785fd9942d6b25d295ed79ba45f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/77c22e51acea433ba2a4a7898423be1d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e12ec6f3fc9c423aa758603f0e096ed4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649] to archive 2023-03-24 01:55:39,726 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:55:39,727 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/5ce915cfcb564991b6d64543d91f7ae4 2023-03-24 01:55:39,728 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2dd1d65a63974fbab3a152c419df306b 2023-03-24 01:55:39,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/fdf17f4593954afdb5fc8f7092184df7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/fdf17f4593954afdb5fc8f7092184df7 2023-03-24 01:55:39,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a97cadd60ba34f43965872a2e8101165 2023-03-24 01:55:39,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a918c73dceaa48bca676d0e6f436012a 2023-03-24 01:55:39,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/0daba68d37ed49af94ec19cc2c9d7134 2023-03-24 01:55:39,736 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cc97830cd3ea4aeaa1a3a9180b444605 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cc97830cd3ea4aeaa1a3a9180b444605 2023-03-24 01:55:39,737 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6d4a66e63504c79a6d9431bac0d65b5 2023-03-24 01:55:39,738 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/cf18196a6360442b99eeab037f7146df 2023-03-24 01:55:39,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/354fd785fd9942d6b25d295ed79ba45f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/354fd785fd9942d6b25d295ed79ba45f 2023-03-24 01:55:39,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7f487cd1ad0649f3936d1459e6c344eb 2023-03-24 01:55:39,743 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/a115d4086a3547b9a5aa97817a177006 2023-03-24 01:55:39,744 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/77c22e51acea433ba2a4a7898423be1d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/77c22e51acea433ba2a4a7898423be1d 2023-03-24 01:55:39,745 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e2d5baff3dc14714b399a230a62524b9 2023-03-24 01:55:39,746 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d6706695bb9e4174b9741b2b9360b691 2023-03-24 01:55:39,746 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f211bef68e7140169e610fd6a8c7b50b 2023-03-24 01:55:39,750 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/17f8991a220642f6afe1f4b2f98f8664 2023-03-24 01:55:39,752 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/106663453846401dad0433adcc5a96c3 2023-03-24 01:55:39,753 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ac37f06b85b843be9407d139b40600ab 2023-03-24 01:55:39,754 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e12ec6f3fc9c423aa758603f0e096ed4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/e12ec6f3fc9c423aa758603f0e096ed4 2023-03-24 01:55:39,755 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/babae0cc05cc4d6e864925c080e6c952 2023-03-24 01:55:39,756 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/2a3dd718d0e5482692708b79edc95f48 2023-03-24 01:55:39,757 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/d3c28cc18130447097ac18a7a72462e9 2023-03-24 01:55:39,758 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/58e731e88ec3453ea09c252e01c86649 2023-03-24 01:55:39,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679622999841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679622999848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679622999849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:39,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1679622999855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,860 DEBUG [Thread-2328] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:55:39,938 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/25675c3d710045238448f75ac38467fb 2023-03-24 01:55:39,948 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/48cb5aa6bdf14747bb6d98c471941f69 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/48cb5aa6bdf14747bb6d98c471941f69 2023-03-24 01:55:39,952 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/48cb5aa6bdf14747bb6d98c471941f69, entries=150, sequenceid=435, filesize=25.4 K 2023-03-24 01:55:39,954 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4236014db93d44d38d9f678f5ed49d79 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4236014db93d44d38d9f678f5ed49d79 2023-03-24 01:55:39,957 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4236014db93d44d38d9f678f5ed49d79, entries=150, sequenceid=435, filesize=11.9 K 2023-03-24 01:55:39,958 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/25675c3d710045238448f75ac38467fb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/25675c3d710045238448f75ac38467fb 2023-03-24 01:55:39,961 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/25675c3d710045238448f75ac38467fb, entries=150, sequenceid=435, filesize=11.9 K 2023-03-24 01:55:39,962 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a5a2c70f4dec9fd5af77e99de4acd24f in 1271ms, sequenceid=435, compaction requested=true 2023-03-24 01:55:39,962 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:39,962 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:39,962 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:39,973 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,973 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:39,974 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,974 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:39,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:39,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b264fd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:39,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:39,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,978 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:39,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:39,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:39,979 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,979 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:39,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:39,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:39,980 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,980 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:39,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:39,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:39,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:39,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:39,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:39,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:39,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:39,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:39,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:39,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:39,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:39,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:39,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:39,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:39,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:39,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:39,994 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:39,994 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:39,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:40,094 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:40,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:40,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:40,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:40,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:40,098 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:40,098 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:40,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:40,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:40,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:40,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:40,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:40,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:40,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:40,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:40,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:40,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:40,103 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:55:40,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:40,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:40,104 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:40,104 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:40,104 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:40,104 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:40,115 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a8d84043c91f4f5f9a3b34e4444eb4bd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a8d84043c91f4f5f9a3b34e4444eb4bd 2023-03-24 01:55:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:40,148 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:40,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679623000162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679623000163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679623000168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679623000269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679623000269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679623000272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679623000475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679623000476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679623000477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,524 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=455, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/9655f943f9f144a78ffbf66069b5301c 2023-03-24 01:55:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56648 deadline: 1679623000781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56622 deadline: 1679623000781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:40,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56680 deadline: 1679623000782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:40,951 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/e1156f94f28846b0866d4188687318a5 2023-03-24 01:55:40,996 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ff7512e23e6f4f8da4a2fba8c753594d 2023-03-24 01:55:41,000 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/9655f943f9f144a78ffbf66069b5301c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9655f943f9f144a78ffbf66069b5301c 2023-03-24 01:55:41,005 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9655f943f9f144a78ffbf66069b5301c, entries=150, sequenceid=455, filesize=25.4 K 2023-03-24 01:55:41,006 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/e1156f94f28846b0866d4188687318a5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/e1156f94f28846b0866d4188687318a5 2023-03-24 01:55:41,010 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/e1156f94f28846b0866d4188687318a5, entries=150, sequenceid=455, filesize=11.9 K 2023-03-24 01:55:41,010 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ff7512e23e6f4f8da4a2fba8c753594d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ff7512e23e6f4f8da4a2fba8c753594d 2023-03-24 01:55:41,014 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ff7512e23e6f4f8da4a2fba8c753594d, entries=150, sequenceid=455, filesize=11.9 K 2023-03-24 01:55:41,015 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a5a2c70f4dec9fd5af77e99de4acd24f in 912ms, sequenceid=455, compaction requested=true 2023-03-24 01:55:41,015 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:41,015 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:41,015 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:41,015 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:41,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:41,016 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:41,018 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca2944a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:41,020 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:41,020 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:41,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:41,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,022 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:41,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:41,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:41,030 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:41,031 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:41,031 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:41,031 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:41,031 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:41,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:41,034 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,035 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,036 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:41,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:41,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:41,043 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:41,044 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:41,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:41,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:41,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:41,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:41,045 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:41,045 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:41,145 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:41,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:41,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:41,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:41,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:41,148 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:41,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:41,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:41,148 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:41,149 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,149 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:41,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:41,150 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:41,151 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:41,151 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:41,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:41,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:41,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:41,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:41,152 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:41,152 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. started... 2023-03-24 01:55:41,152 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:55:41,152 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:41,152 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:41,153 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:41,153 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:41,153 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:41,153 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:41,191 DEBUG [Thread-2343] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:62686 2023-03-24 01:55:41,191 DEBUG [Thread-2343] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,192 DEBUG [Thread-2345] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:62686 2023-03-24 01:55:41,192 DEBUG [Thread-2339] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:62686 2023-03-24 01:55:41,192 DEBUG [Thread-2345] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,192 DEBUG [Thread-2339] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,197 DEBUG [Thread-2341] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:62686 2023-03-24 01:55:41,197 DEBUG [Thread-2341] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,197 DEBUG [Thread-2337] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:62686 2023-03-24 01:55:41,197 DEBUG [Thread-2337] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:41,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. as already flushing 2023-03-24 01:55:41,288 DEBUG [Thread-2330] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:62686 2023-03-24 01:55:41,288 DEBUG [Thread-2330] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,288 DEBUG [Thread-2334] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:62686 2023-03-24 01:55:41,288 DEBUG [Thread-2334] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,292 DEBUG [Thread-2332] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:62686 2023-03-24 01:55:41,292 DEBUG [Thread-2332] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:41,565 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240495d3a495054f5689b4221e0ffaecc8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240495d3a495054f5689b4221e0ffaecc8 2023-03-24 01:55:41,970 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/93ba6f75672a47bdacfe694287423fb8 2023-03-24 01:55:42,381 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/80450c1ff8cc43548dd535ac74284b2d 2023-03-24 01:55:42,790 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/7540009157bc4daa992cf438d566dbcd 2023-03-24 01:55:42,793 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/93ba6f75672a47bdacfe694287423fb8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/93ba6f75672a47bdacfe694287423fb8 2023-03-24 01:55:42,796 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/93ba6f75672a47bdacfe694287423fb8, entries=150, sequenceid=467, filesize=25.4 K 2023-03-24 01:55:42,796 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/80450c1ff8cc43548dd535ac74284b2d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/80450c1ff8cc43548dd535ac74284b2d 2023-03-24 01:55:42,799 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/80450c1ff8cc43548dd535ac74284b2d, entries=150, sequenceid=467, filesize=11.9 K 2023-03-24 01:55:42,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/7540009157bc4daa992cf438d566dbcd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7540009157bc4daa992cf438d566dbcd 2023-03-24 01:55:42,802 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7540009157bc4daa992cf438d566dbcd, entries=150, sequenceid=467, filesize=11.9 K 2023-03-24 01:55:42,802 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for a5a2c70f4dec9fd5af77e99de4acd24f in 1650ms, sequenceid=467, compaction requested=true 2023-03-24 01:55:42,802 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:42,803 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:42,803 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,804 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:42,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:42,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:42,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a98476[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:42,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:42,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,807 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:42,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:42,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:42,807 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,807 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:42,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:42,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:42,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:42,810 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:42,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:42,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:42,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:42,810 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:42,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:42,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:42,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,814 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:42,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:42,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:42,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:42,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:42,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:42,816 DEBUG [Thread-2336] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:42,816 DEBUG [Thread-2336] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:42,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:42,916 DEBUG [Thread-2336] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:42,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:47,399 DEBUG [Thread-2326] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:62686 2023-03-24 01:55:47,399 DEBUG [Thread-2326] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:49,865 DEBUG [Thread-2328] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:62686 2023-03-24 01:55:49,865 DEBUG [Thread-2328] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 70 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1738 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5211 rows 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1764 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5291 rows 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1752 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5256 rows 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1745 2023-03-24 01:55:49,865 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5233 rows 2023-03-24 01:55:49,866 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1765 2023-03-24 01:55:49,866 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5293 rows 2023-03-24 01:55:49,866 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:55:49,866 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:62686 2023-03-24 01:55:49,866 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:55:49,869 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:55:49,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:55:49,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:49,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:49,874 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622949874"}]},"ts":"1679622949874"} 2023-03-24 01:55:49,874 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:55:49,877 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:55:49,877 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, UNASSIGN}] 2023-03-24 01:55:49,879 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, UNASSIGN 2023-03-24 01:55:49,879 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:49,879 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622949879"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622949879"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622949879"}]},"ts":"1679622949879"} 2023-03-24 01:55:49,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:49,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:50,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:50,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a5a2c70f4dec9fd5af77e99de4acd24f, disabling compactions & flushes 2023-03-24 01:55:50,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. after waiting 0 ms 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:50,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a5a2c70f4dec9fd5af77e99de4acd24f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=A 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=B 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5a2c70f4dec9fd5af77e99de4acd24f, store=C 2023-03-24 01:55:50,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:50,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032412c3877a97e8420d9c7162506dbd09f0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032412c3877a97e8420d9c7162506dbd09f0 2023-03-24 01:55:50,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:50,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/3d0d75f7a98741e68c3264483e59e18d 2023-03-24 01:55:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:51,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4d78380fdb68417b916f7ddbd33a49cb 2023-03-24 01:55:51,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ea212cd9c51f456d9ab126c05e33cd44 2023-03-24 01:55:51,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/A/3d0d75f7a98741e68c3264483e59e18d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/3d0d75f7a98741e68c3264483e59e18d 2023-03-24 01:55:51,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/3d0d75f7a98741e68c3264483e59e18d, entries=150, sequenceid=475, filesize=25.4 K 2023-03-24 01:55:51,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/B/4d78380fdb68417b916f7ddbd33a49cb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4d78380fdb68417b916f7ddbd33a49cb 2023-03-24 01:55:51,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4d78380fdb68417b916f7ddbd33a49cb, entries=150, sequenceid=475, filesize=11.9 K 2023-03-24 01:55:51,673 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/.tmp/C/ea212cd9c51f456d9ab126c05e33cd44 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ea212cd9c51f456d9ab126c05e33cd44 2023-03-24 01:55:51,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ea212cd9c51f456d9ab126c05e33cd44, entries=150, sequenceid=475, filesize=11.9 K 2023-03-24 01:55:51,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for a5a2c70f4dec9fd5af77e99de4acd24f in 1644ms, sequenceid=475, compaction requested=true 2023-03-24 01:55:51,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits/478.seqid, newMaxSeqId=478, maxSeqId=4 2023-03-24 01:55:51,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f. 2023-03-24 01:55:51,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a5a2c70f4dec9fd5af77e99de4acd24f: 2023-03-24 01:55:51,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:51,682 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a5a2c70f4dec9fd5af77e99de4acd24f, regionState=CLOSED 2023-03-24 01:55:51,682 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622951682"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622951682"}]},"ts":"1679622951682"} 2023-03-24 01:55:51,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-24 01:55:51,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure a5a2c70f4dec9fd5af77e99de4acd24f, server=jenkins-hbase4.apache.org,44439,1679622817494 in 1.8030 sec 2023-03-24 01:55:51,685 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-24 01:55:51,685 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5a2c70f4dec9fd5af77e99de4acd24f, UNASSIGN in 1.8070 sec 2023-03-24 01:55:51,686 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622951686"}]},"ts":"1679622951686"} 2023-03-24 01:55:51,687 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:55:51,688 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:55:51,689 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8190 sec 2023-03-24 01:55:51,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 01:55:51,977 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-24 01:55:51,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:55:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:51,980 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 01:55:51,980 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:51,985 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:51,986 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits] 2023-03-24 01:55:51,989 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/3d0d75f7a98741e68c3264483e59e18d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/3d0d75f7a98741e68c3264483e59e18d 2023-03-24 01:55:51,990 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/48cb5aa6bdf14747bb6d98c471941f69 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/48cb5aa6bdf14747bb6d98c471941f69 2023-03-24 01:55:51,991 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/93ba6f75672a47bdacfe694287423fb8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/93ba6f75672a47bdacfe694287423fb8 2023-03-24 01:55:51,991 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9655f943f9f144a78ffbf66069b5301c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/9655f943f9f144a78ffbf66069b5301c 2023-03-24 01:55:51,992 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a053b08a36e844028008bfbd38c324e6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a053b08a36e844028008bfbd38c324e6 2023-03-24 01:55:51,993 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a8990f3513ff44dcad8e7957f2ac79b4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/A/a8990f3513ff44dcad8e7957f2ac79b4 2023-03-24 01:55:51,996 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4236014db93d44d38d9f678f5ed49d79 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4236014db93d44d38d9f678f5ed49d79 2023-03-24 01:55:51,997 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4d78380fdb68417b916f7ddbd33a49cb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/4d78380fdb68417b916f7ddbd33a49cb 2023-03-24 01:55:51,998 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/503e22bea8334f3791b402ba05bb151e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/503e22bea8334f3791b402ba05bb151e 2023-03-24 01:55:51,999 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/80450c1ff8cc43548dd535ac74284b2d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/80450c1ff8cc43548dd535ac74284b2d 2023-03-24 01:55:52,000 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/e1156f94f28846b0866d4188687318a5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/e1156f94f28846b0866d4188687318a5 2023-03-24 01:55:52,001 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/f91c8901b3c94191bf09cc48b6fc714c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/B/f91c8901b3c94191bf09cc48b6fc714c 2023-03-24 01:55:52,003 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/25675c3d710045238448f75ac38467fb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/25675c3d710045238448f75ac38467fb 2023-03-24 01:55:52,004 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/3ee4614becb94460921430f603395039 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/3ee4614becb94460921430f603395039 2023-03-24 01:55:52,005 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7540009157bc4daa992cf438d566dbcd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/7540009157bc4daa992cf438d566dbcd 2023-03-24 01:55:52,006 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ea212cd9c51f456d9ab126c05e33cd44 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ea212cd9c51f456d9ab126c05e33cd44 2023-03-24 01:55:52,008 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f8c39badccde49eda7f6d9729d820774 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/f8c39badccde49eda7f6d9729d820774 2023-03-24 01:55:52,009 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ff7512e23e6f4f8da4a2fba8c753594d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/C/ff7512e23e6f4f8da4a2fba8c753594d 2023-03-24 01:55:52,011 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits/478.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f/recovered.edits/478.seqid 2023-03-24 01:55:52,011 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/a5a2c70f4dec9fd5af77e99de4acd24f 2023-03-24 01:55:52,011 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:55:52,011 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:55:52,012 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 01:55:52,014 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400df78b9e91d44a190960fc4a690b2b0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032400df78b9e91d44a190960fc4a690b2b0 2023-03-24 01:55:52,015 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032402ce4d2eebae4de79150bd2c9188d37c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032402ce4d2eebae4de79150bd2c9188d37c 2023-03-24 01:55:52,016 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240495d3a495054f5689b4221e0ffaecc8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240495d3a495054f5689b4221e0ffaecc8 2023-03-24 01:55:52,017 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032412c3877a97e8420d9c7162506dbd09f0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032412c3877a97e8420d9c7162506dbd09f0 2023-03-24 01:55:52,018 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032415f02af9b8814fc69eda168d70b105ee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032415f02af9b8814fc69eda168d70b105ee 2023-03-24 01:55:52,019 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032416ab429191b6418eaef7ccf35e8be96e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032416ab429191b6418eaef7ccf35e8be96e 2023-03-24 01:55:52,020 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032423fa023f25b8448d83a1f1e1ffb10163 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032423fa023f25b8448d83a1f1e1ffb10163 2023-03-24 01:55:52,020 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a4097f67dd34429a798c4785b84cfbc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a4097f67dd34429a798c4785b84cfbc 2023-03-24 01:55:52,021 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242af61bb2960b4c6bb92fdfc08de74ec8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242af61bb2960b4c6bb92fdfc08de74ec8 2023-03-24 01:55:52,022 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a1445d1eee647b6af6c679d45aa418e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303243a1445d1eee647b6af6c679d45aa418e 2023-03-24 01:55:52,023 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244d7b3d0d99ea4c999e7eac2f5aa6dca8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244d7b3d0d99ea4c999e7eac2f5aa6dca8 2023-03-24 01:55:52,024 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245878230626ee4a62b368daf7b49c8a9c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245878230626ee4a62b368daf7b49c8a9c 2023-03-24 01:55:52,024 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245958be374b86495087a02342d80cfd60 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245958be374b86495087a02342d80cfd60 2023-03-24 01:55:52,025 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461508d0964f94d1ebea886e47ceb196e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461508d0964f94d1ebea886e47ceb196e 2023-03-24 01:55:52,026 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032490307f9fb8254b9ea45bb761a335c5d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032490307f9fb8254b9ea45bb761a335c5d3 2023-03-24 01:55:52,027 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249404401014694b3ca0aee6f2a3efe85a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249404401014694b3ca0aee6f2a3efe85a 2023-03-24 01:55:52,028 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a8d84043c91f4f5f9a3b34e4444eb4bd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a8d84043c91f4f5f9a3b34e4444eb4bd 2023-03-24 01:55:52,029 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c053550f9e77482e964dc34aab37614a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c053550f9e77482e964dc34aab37614a 2023-03-24 01:55:52,029 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ce1056f9af224a7f831662b7d84df85f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ce1056f9af224a7f831662b7d84df85f 2023-03-24 01:55:52,030 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d0a6cfd03a3540e985185f0e3d929653 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d0a6cfd03a3540e985185f0e3d929653 2023-03-24 01:55:52,031 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dbaa04b355654b2c82c7f4ee2d0b4775 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dbaa04b355654b2c82c7f4ee2d0b4775 2023-03-24 01:55:52,031 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3d8390d460e4a7995d32e411a704daf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e3d8390d460e4a7995d32e411a704daf 2023-03-24 01:55:52,032 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e82bbcf1901d496ab7da39db6e0160ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e82bbcf1901d496ab7da39db6e0160ab 2023-03-24 01:55:52,033 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f926bb5e8b3f4448823a26ba7e46edda to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f926bb5e8b3f4448823a26ba7e46edda 2023-03-24 01:55:52,033 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:55:52,035 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:52,036 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:55:52,038 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:55:52,038 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:52,038 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:55:52,038 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679622952038"}]},"ts":"9223372036854775807"} 2023-03-24 01:55:52,040 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:55:52,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a5a2c70f4dec9fd5af77e99de4acd24f, NAME => 'TestAcidGuarantees,,1679622918121.a5a2c70f4dec9fd5af77e99de4acd24f.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:55:52,040 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:55:52,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679622952040"}]},"ts":"9223372036854775807"} 2023-03-24 01:55:52,041 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:55:52,042 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:52,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-03-24 01:55:52,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 01:55:52,081 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-24 01:55:52,092 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 228), OpenFileDescriptor=509 (was 563), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 300) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 172), AvailableMemoryMB=5370 (was 5414) 2023-03-24 01:55:52,103 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=509, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=171, AvailableMemoryMB=5370 2023-03-24 01:55:52,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:55:52,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:55:52,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:55:52,107 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:55:52,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-24 01:55:52,107 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:55:52,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:52,109 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:52,109 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 empty. 2023-03-24 01:55:52,109 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:52,109 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:55:52,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:52,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:52,516 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:55:52,518 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7db42890c5a058f4b49bb0a2fb398589, NAME => 'TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:55:52,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7db42890c5a058f4b49bb0a2fb398589, disabling compactions & flushes 2023-03-24 01:55:52,934 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. after waiting 0 ms 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:52,934 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:52,934 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:52,936 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:55:52,937 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622952937"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622952937"}]},"ts":"1679622952937"} 2023-03-24 01:55:52,938 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:55:52,939 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:55:52,939 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622952939"}]},"ts":"1679622952939"} 2023-03-24 01:55:52,939 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:55:52,943 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, ASSIGN}] 2023-03-24 01:55:52,944 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, ASSIGN 2023-03-24 01:55:52,945 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:55:53,096 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=7db42890c5a058f4b49bb0a2fb398589, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:53,096 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622953096"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622953096"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622953096"}]},"ts":"1679622953096"} 2023-03-24 01:55:53,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:55:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:53,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:53,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7db42890c5a058f4b49bb0a2fb398589, NAME => 'TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:55:53,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:55:53,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,253 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,254 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A 2023-03-24 01:55:53,254 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A 2023-03-24 01:55:53,254 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:53,255 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7db42890c5a058f4b49bb0a2fb398589 columnFamilyName A 2023-03-24 01:55:53,255 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(310): Store=7db42890c5a058f4b49bb0a2fb398589/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:53,255 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,256 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B 2023-03-24 01:55:53,256 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B 2023-03-24 01:55:53,256 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:53,256 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7db42890c5a058f4b49bb0a2fb398589 columnFamilyName B 2023-03-24 01:55:53,257 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(310): Store=7db42890c5a058f4b49bb0a2fb398589/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:53,257 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,258 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C 2023-03-24 01:55:53,258 DEBUG [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C 2023-03-24 01:55:53,258 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:55:53,258 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7db42890c5a058f4b49bb0a2fb398589 columnFamilyName C 2023-03-24 01:55:53,258 INFO [StoreOpener-7db42890c5a058f4b49bb0a2fb398589-1] regionserver.HStore(310): Store=7db42890c5a058f4b49bb0a2fb398589/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:55:53,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:53,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:55:53,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:53,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:55:53,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7db42890c5a058f4b49bb0a2fb398589; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61183777, jitterRate=-0.08829067647457123}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:55:53,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:53,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., pid=49, masterSystemTime=1679622953249 2023-03-24 01:55:53,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:53,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:53,265 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=7db42890c5a058f4b49bb0a2fb398589, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:53,265 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622953265"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622953265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622953265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622953265"}]},"ts":"1679622953265"} 2023-03-24 01:55:53,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-24 01:55:53,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 in 169 msec 2023-03-24 01:55:53,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-24 01:55:53,268 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, ASSIGN in 324 msec 2023-03-24 01:55:53,269 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:55:53,269 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622953269"}]},"ts":"1679622953269"} 2023-03-24 01:55:53,270 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:55:53,271 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:55:53,272 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1660 sec 2023-03-24 01:55:54,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 01:55:54,212 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-24 01:55:54,213 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x681750df to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,218 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e372785, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,219 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,221 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,222 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:55:54,223 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40206, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:55:54,224 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x02df63a4 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,228 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@501c5a88, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,228 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,232 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,232 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,236 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,236 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,240 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,240 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,245 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,245 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,249 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,250 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x461f5f3f to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,254 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@426cb82a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,254 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,259 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,259 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,263 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,263 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:55:54,267 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:55:54,271 DEBUG [hconnection-0x44f209d1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,272 DEBUG [hconnection-0x35bc092b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,272 DEBUG [hconnection-0x25edeefb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,272 DEBUG [hconnection-0x136fd94e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,273 DEBUG [hconnection-0x773170b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,273 DEBUG [hconnection-0x7238d4fa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,273 DEBUG [hconnection-0x1948d252-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,273 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35014, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,273 DEBUG [hconnection-0x5ffa6545-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,274 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35048, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,274 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35074, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,276 DEBUG [hconnection-0x6944eae5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,277 DEBUG [hconnection-0x5a62e14d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:55:54,281 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,281 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34998, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,281 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35036, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,281 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35070, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,282 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35054, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,285 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35086, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,286 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35084, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:55:54,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,293 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,294 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:54,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:55:54,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,295 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:54,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:54,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,296 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,297 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,297 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,300 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,300 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@686bbbc6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,304 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,306 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,307 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,309 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,309 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,309 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,311 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,311 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,311 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/58a4422791a54a7c89c5fc95610478f1 2023-03-24 01:55:54,324 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622954315, max timestamp=1679622954325Num uniques 859; 2023-03-24 01:55:54,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:55:54,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622954315, max timestamp=1679622954325Num uniques 859; 2023-03-24 01:55:54,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:55:54,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622954322, max timestamp=1679622954325Num uniques 859; 2023-03-24 01:55:54,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:54,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,333 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,333 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623014337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623014337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623014338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623014338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623014338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,433 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:54,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:54,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623014439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623014439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623014439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623014440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623014440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,444 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,448 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395afba[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,451 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,452 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,452 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,452 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,455 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,457 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,457 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,463 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,462 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,463 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,463 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,563 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:54,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,568 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,570 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,570 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,570 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,570 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,572 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,575 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bad6b46[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,577 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,577 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,578 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,587 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,587 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,593 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:54,593 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,593 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623014642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623014643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623014643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623014643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623014644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,694 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:54,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,696 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,696 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,700 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,700 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,700 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,700 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,701 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6579d271[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,703 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,706 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,708 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,708 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,710 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,710 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,711 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,714 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,714 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,714 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/467b7b3fb87a475b8d0666afe35895db 2023-03-24 01:55:54,814 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:54,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:54,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,821 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,821 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,826 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,826 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,828 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f9e093[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,833 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,833 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,839 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,839 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,845 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,848 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,848 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:54,949 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:54,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:54,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623014949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:54,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:54,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:54,953 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:54,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:54,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:54,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:54,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:54,955 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:54,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:54,956 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:54,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:54,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623014949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,957 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,957 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623014951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:54,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623014951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623014951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:54,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebf7d49[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:54,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:54,959 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,961 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:54,962 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,962 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:54,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,963 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:54,965 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:54,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:54,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:54,965 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:54,965 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:54,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,967 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:54,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,968 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,969 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:54,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:54,971 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:54,971 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:54,972 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:54,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:55,072 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:55,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:55,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:55,075 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:55,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:55,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:55,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:55,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:55,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:55,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,078 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:55,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:55,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:55,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e18ce83[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:55,082 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:55,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:55,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:55,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:55,087 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:55,087 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:55,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:55,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:55,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:55,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:55,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:55,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:55,098 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:55,098 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:55,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:55,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:55,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:55,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/76dcc33093554c0fa17ef347c09287ab 2023-03-24 01:55:55,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/58a4422791a54a7c89c5fc95610478f1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1 2023-03-24 01:55:55,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1, entries=250, sequenceid=11, filesize=16.2 K 2023-03-24 01:55:55,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/467b7b3fb87a475b8d0666afe35895db as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db 2023-03-24 01:55:55,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db, entries=150, sequenceid=11, filesize=11.6 K 2023-03-24 01:55:55,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/76dcc33093554c0fa17ef347c09287ab as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab 2023-03-24 01:55:55,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab, entries=150, sequenceid=11, filesize=11.6 K 2023-03-24 01:55:55,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 7db42890c5a058f4b49bb0a2fb398589 in 892ms, sequenceid=11, compaction requested=false 2023-03-24 01:55:55,188 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 01:55:55,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:55,198 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:55,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:55,201 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:55,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:55,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:55,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,204 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:55,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:55,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:55,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:55,205 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:55,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:55,206 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:55,206 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:55,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:55,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:55,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623015467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623015468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623015469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623015469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623015472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623015573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623015579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623015579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623015579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623015579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,629 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f67166ba6d31470bbade6399a134080e 2023-03-24 01:55:55,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623015780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623015784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623015784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623015784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623015785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,039 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/b416c03c6aa64b1ebab5ebd519cee7d6 2023-03-24 01:55:56,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623016083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623016092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623016092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623016093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623016094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,448 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/b620bc81f8244377b9894aaf8afe6ebe 2023-03-24 01:55:56,452 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f67166ba6d31470bbade6399a134080e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e 2023-03-24 01:55:56,457 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e, entries=150, sequenceid=47, filesize=11.6 K 2023-03-24 01:55:56,458 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/b416c03c6aa64b1ebab5ebd519cee7d6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6 2023-03-24 01:55:56,461 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6, entries=150, sequenceid=47, filesize=11.6 K 2023-03-24 01:55:56,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/b620bc81f8244377b9894aaf8afe6ebe as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe 2023-03-24 01:55:56,465 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe, entries=150, sequenceid=47, filesize=11.6 K 2023-03-24 01:55:56,466 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 7db42890c5a058f4b49bb0a2fb398589 in 1261ms, sequenceid=47, compaction requested=false 2023-03-24 01:55:56,469 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:56,469 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,475 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5596b6b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:56,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:56,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:56,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:56,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:56,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:56,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:56,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:56,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:56,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:56,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:56,484 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:56,486 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,490 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,490 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:56,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,491 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:56,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:56,492 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:56,492 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:56,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:56,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:56,592 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:56,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:56,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:56,596 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:56,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:56,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:56,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,596 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:56,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:56,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/792c7e3e2af24f42ad81c5553ac7c452 2023-03-24 01:55:56,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,602 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:56,603 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,603 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:56,603 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:56,603 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:56,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3bee2b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:56,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:56,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:56,613 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:56,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:56,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:56,620 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:56,620 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:56,620 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:56,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:56,621 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:56,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/88bae949e5784f368dda9a465fdfdb8d 2023-03-24 01:55:56,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,623 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,624 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,627 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:56,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,627 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:56,627 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:56,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:56,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622956617, max timestamp=1679622956628Num uniques 859; 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622956617, max timestamp=1679622956628Num uniques 859; 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622956617, max timestamp=1679622956628Num uniques 859; 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:56,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:56,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623016662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622956649, max timestamp=1679622956692Num uniques 1018; 2023-03-24 01:55:56,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:55:56,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622956649, max timestamp=1679622956692Num uniques 1018; 2023-03-24 01:55:56,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:55:56,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622956649, max timestamp=1679622956692Num uniques 1018; 2023-03-24 01:55:56,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:55:56,728 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:56,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:56,731 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:56,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:56,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:56,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,733 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,733 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,734 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:56,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623016726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623016728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:56,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:56,735 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,735 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:56,735 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:56,735 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:56,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623016731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6b9d10[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:56,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:56,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:56,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:56,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:56,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623016734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,742 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:56,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:56,745 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:56,745 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:56,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:56,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:56,746 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:56,747 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,747 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,747 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,748 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:56,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:56,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,755 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:56,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,755 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:56,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623016795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623016835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623016835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623016837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:56,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623016842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,856 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:56,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:56,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:56,859 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:56,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:56,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:56,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,860 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:56,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:56,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:56,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:56,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:56,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:56,864 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@776f34a8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:56,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:56,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:56,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:56,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:56,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,873 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:56,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:56,876 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:56,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:56,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:56,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:56,876 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:56,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,878 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,879 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:56,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:56,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:56,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,881 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,881 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:56,981 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:56,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:56,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:56,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:56,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:56,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:56,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:56,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,987 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:56,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:56,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:56,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:56,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:56,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:56,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:56,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:56,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:56,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a31b1d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:56,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:56,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,995 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:56,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:56,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:56,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:56,999 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:56,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:56,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:56,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:56,999 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,000 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,001 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,002 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623017000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,005 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,006 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,006 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:57,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/6c545aacdf6b437484162f08a6b574b3 2023-03-24 01:55:57,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/792c7e3e2af24f42ad81c5553ac7c452 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452 2023-03-24 01:55:57,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452, entries=150, sequenceid=57, filesize=11.6 K 2023-03-24 01:55:57,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/88bae949e5784f368dda9a465fdfdb8d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d 2023-03-24 01:55:57,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623017039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623017039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623017042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d, entries=150, sequenceid=57, filesize=11.6 K 2023-03-24 01:55:57,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/6c545aacdf6b437484162f08a6b574b3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3 2023-03-24 01:55:57,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3, entries=150, sequenceid=57, filesize=11.6 K 2023-03-24 01:55:57,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7db42890c5a058f4b49bb0a2fb398589 in 459ms, sequenceid=57, compaction requested=true 2023-03-24 01:55:57,049 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:57,049 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:57,049 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:57,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:57,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:57,050 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:57,050 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:57,050 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:55:57,050 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:55:57,050 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,050 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,050 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=39.4 K 2023-03-24 01:55:57,051 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=34.8 K 2023-03-24 01:55:57,051 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 58a4422791a54a7c89c5fc95610478f1, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679622954283 2023-03-24 01:55:57,051 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 467b7b3fb87a475b8d0666afe35895db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679622954283 2023-03-24 01:55:57,051 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f67166ba6d31470bbade6399a134080e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679622954336 2023-03-24 01:55:57,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:57,055 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b416c03c6aa64b1ebab5ebd519cee7d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679622954336 2023-03-24 01:55:57,055 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 792c7e3e2af24f42ad81c5553ac7c452, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622956724, max timestamp=1679622957052Num uniques 1027; 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622956724, max timestamp=1679622957052Num uniques 1027; 2023-03-24 01:55:57,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=140.89 KB heapSize=404.77 KB 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 01:55:57,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622956724, max timestamp=1679622957052Num uniques 1027; 2023-03-24 01:55:57,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 01:55:57,055 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,055 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:57,055 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 88bae949e5784f368dda9a465fdfdb8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:57,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:57,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,061 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:57,063 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:57,106 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,106 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/221824cef6f543b7a994b652b4006fa2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/221824cef6f543b7a994b652b4006fa2 2023-03-24 01:55:57,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,110 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,110 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,112 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,112 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 221824cef6f543b7a994b652b4006fa2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,112 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,112 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622957049; duration=0sec 2023-03-24 01:55:57,112 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:57,112 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,114 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:57,114 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:55:57,114 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,114 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=34.8 K 2023-03-24 01:55:57,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,115 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 76dcc33093554c0fa17ef347c09287ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679622954283 2023-03-24 01:55:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,116 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,116 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,116 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,116 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,116 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b620bc81f8244377b9894aaf8afe6ebe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1679622954336 2023-03-24 01:55:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:57,116 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6c545aacdf6b437484162f08a6b574b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:57,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:57,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cace0b4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:57,119 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:57,120 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,121 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:57,122 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f1eb79572a48463eb3e0000b34b1d074 2023-03-24 01:55:57,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:57,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,123 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:57,124 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:57,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:57,124 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,137 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,141 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:57,142 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,142 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,142 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,142 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:57,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/4b6ae166392c48c99df5cac4618dc8b0 2023-03-24 01:55:57,154 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/a4822353da624dcebdcd99b06d2e67ab as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a4822353da624dcebdcd99b06d2e67ab 2023-03-24 01:55:57,157 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into a4822353da624dcebdcd99b06d2e67ab(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:57,157 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,157 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622957050; duration=0sec 2023-03-24 01:55:57,157 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622957148, max timestamp=1679622957183Num uniques 859; 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622957148, max timestamp=1679622957183Num uniques 859; 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622957148, max timestamp=1679622957183Num uniques 859; 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:57,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:55:57,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/143c204805704d93a5ea1a6c9cc2e25e 2023-03-24 01:55:57,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f1eb79572a48463eb3e0000b34b1d074 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074 2023-03-24 01:55:57,207 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:55:57,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074, entries=200, sequenceid=112, filesize=13.9 K 2023-03-24 01:55:57,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/4b6ae166392c48c99df5cac4618dc8b0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0 2023-03-24 01:55:57,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0, entries=150, sequenceid=112, filesize=11.6 K 2023-03-24 01:55:57,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/143c204805704d93a5ea1a6c9cc2e25e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e 2023-03-24 01:55:57,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e, entries=150, sequenceid=112, filesize=11.6 K 2023-03-24 01:55:57,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~64.50 KB/66048, currentSize=73.80 KB/75570 for 7db42890c5a058f4b49bb0a2fb398589 in 184ms, sequenceid=112, compaction requested=false 2023-03-24 01:55:57,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,243 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=80.51 KB heapSize=225.61 KB 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:57,249 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:55:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:57,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,256 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,258 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,258 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:57,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fec712c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:57,276 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:57,276 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:57,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:57,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:57,278 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,278 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:57,279 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,279 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:57,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:57,282 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:57,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:57,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:57,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:57,282 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/feaf7ed6efdf474ea240b11f8e793745 2023-03-24 01:55:57,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,288 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,295 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,295 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,295 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,295 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623017337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623017343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623017347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623017348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623017348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,396 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,399 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,403 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,403 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,403 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,403 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:57,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:57,406 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:57,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31a8d80d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:57,408 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,411 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:57,411 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,411 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:57,412 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:57,414 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:57,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:57,415 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,418 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,420 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:57,421 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,421 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,421 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623017445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623017446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,508 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/4ad85f30f9194fe796c487d2b83dfe79 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4ad85f30f9194fe796c487d2b83dfe79 2023-03-24 01:55:57,512 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 4ad85f30f9194fe796c487d2b83dfe79(size=11.7 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:57,513 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,513 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622957049; duration=0sec 2023-03-24 01:55:57,513 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:57,521 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,524 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,524 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,528 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,528 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,528 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,529 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:57,529 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:57,530 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:57,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f256d6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:57,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:57,531 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,532 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:57,532 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:57,532 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,533 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:57,534 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,534 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:57,536 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:57,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:57,536 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:57,545 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,545 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,544 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,544 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,545 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,645 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,648 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,649 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,650 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,650 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,655 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,655 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,656 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623017651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,656 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:57,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623017651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:57,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b40bd3c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:57,659 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:57,660 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,662 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:57,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:57,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:57,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:57,664 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:57,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:57,664 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:57,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:57,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:57,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,665 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,666 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,667 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,678 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:57,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:57,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:57,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:57,679 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:57,679 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:57,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:57,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e97bb21fb74c4e3b93e582552e9331ea 2023-03-24 01:55:57,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/813c33a63c3c4e66b1de8deb64e3071e 2023-03-24 01:55:57,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/feaf7ed6efdf474ea240b11f8e793745 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745 2023-03-24 01:55:57,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745, entries=200, sequenceid=143, filesize=14.1 K 2023-03-24 01:55:57,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e97bb21fb74c4e3b93e582552e9331ea as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea 2023-03-24 01:55:57,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea, entries=150, sequenceid=143, filesize=11.8 K 2023-03-24 01:55:57,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/813c33a63c3c4e66b1de8deb64e3071e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e 2023-03-24 01:55:57,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e, entries=150, sequenceid=143, filesize=11.8 K 2023-03-24 01:55:57,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~225.56 KB/230976, currentSize=100.63 KB/103050 for 7db42890c5a058f4b49bb0a2fb398589 in 513ms, sequenceid=143, compaction requested=true 2023-03-24 01:55:57,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:57,762 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:57,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:55:57,764 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:57,766 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40641 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:57,766 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:55:57,766 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,766 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4ad85f30f9194fe796c487d2b83dfe79, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=39.7 K 2023-03-24 01:55:57,766 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4ad85f30f9194fe796c487d2b83dfe79, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:57,767 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f1eb79572a48463eb3e0000b34b1d074, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679622956724 2023-03-24 01:55:57,767 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:57,767 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:55:57,767 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,767 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/221824cef6f543b7a994b652b4006fa2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=35.1 K 2023-03-24 01:55:57,767 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting feaf7ed6efdf474ea240b11f8e793745, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:55:57,768 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 221824cef6f543b7a994b652b4006fa2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:57,768 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4b6ae166392c48c99df5cac4618dc8b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679622956725 2023-03-24 01:55:57,769 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e97bb21fb74c4e3b93e582552e9331ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:55:57,777 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:57,778 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:57,779 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:57,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:57,792 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:57,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:57,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,794 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:57,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:57,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:57,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:57,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:57,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:57,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:57,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:57,820 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:57,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:57,837 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/ec78bd9b10a04f8da051d11eab51c624 2023-03-24 01:55:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:57,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:57,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623017886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623017886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623017892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623017957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623017958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:57,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:57,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623017993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623017995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623018001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623018195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623018202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623018208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,223 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/c18be2c5e3b54e0a957ff531235b2c6c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/c18be2c5e3b54e0a957ff531235b2c6c 2023-03-24 01:55:58,224 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/21e3af9d26854ab88bc2b9acf5f03b72 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/21e3af9d26854ab88bc2b9acf5f03b72 2023-03-24 01:55:58,227 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into c18be2c5e3b54e0a957ff531235b2c6c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:58,227 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:58,227 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622957762; duration=0sec 2023-03-24 01:55:58,227 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:55:58,227 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:55:58,228 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 21e3af9d26854ab88bc2b9acf5f03b72(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:58,228 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:58,228 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622957762; duration=0sec 2023-03-24 01:55:58,228 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:58,228 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:55:58,228 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:55:58,229 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:58,229 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a4822353da624dcebdcd99b06d2e67ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=35.1 K 2023-03-24 01:55:58,229 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a4822353da624dcebdcd99b06d2e67ab, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679622955465 2023-03-24 01:55:58,229 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 143c204805704d93a5ea1a6c9cc2e25e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679622956725 2023-03-24 01:55:58,230 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 813c33a63c3c4e66b1de8deb64e3071e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:55:58,235 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:55:58,245 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/c303b96cd2bf45bab4bd3f3f44744e26 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/c303b96cd2bf45bab4bd3f3f44744e26 2023-03-24 01:55:58,248 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into c303b96cd2bf45bab4bd3f3f44744e26(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:55:58,248 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:58,248 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622957762; duration=0sec 2023-03-24 01:55:58,248 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:55:58,266 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/61ecc4d003924826862322785fa6cefa 2023-03-24 01:55:58,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623018464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623018465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623018503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623018509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:58,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623018514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,675 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/9bef35ad07ef460781cb8d81e42e7d03 2023-03-24 01:55:58,679 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/ec78bd9b10a04f8da051d11eab51c624 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624 2023-03-24 01:55:58,682 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624, entries=150, sequenceid=162, filesize=11.8 K 2023-03-24 01:55:58,683 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/61ecc4d003924826862322785fa6cefa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa 2023-03-24 01:55:58,686 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa, entries=150, sequenceid=162, filesize=11.8 K 2023-03-24 01:55:58,687 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/9bef35ad07ef460781cb8d81e42e7d03 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03 2023-03-24 01:55:58,690 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03, entries=150, sequenceid=162, filesize=11.8 K 2023-03-24 01:55:58,691 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7db42890c5a058f4b49bb0a2fb398589 in 871ms, sequenceid=162, compaction requested=false 2023-03-24 01:55:58,691 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:58,691 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:58,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:58,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:58,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:58,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f9c869[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:58,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:58,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,699 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:58,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:58,699 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:58,700 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:58,700 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:58,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:58,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:58,701 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:58,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:58,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:58,701 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:58,701 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:58,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:58,704 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:58,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,704 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:58,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,706 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:58,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:58,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:58,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:58,709 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:58,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:58,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:58,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:58,710 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:58,710 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:58,810 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:58,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:58,813 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:58,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:58,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:58,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:58,813 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,814 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:58,815 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:58,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:58,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:58,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:58,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:58,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:58,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:58,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:58,817 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:58,817 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:58,826 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/34828f41196747e996f5ed4e0b45f884 2023-03-24 01:55:58,844 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/5d94dcee70d147819e26df86cfa19aee 2023-03-24 01:55:59,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:59,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,257 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/6ae80da2b7e04f7abf62063c36f6e10b 2023-03-24 01:55:59,261 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/34828f41196747e996f5ed4e0b45f884 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884 2023-03-24 01:55:59,264 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884, entries=150, sequenceid=179, filesize=11.8 K 2023-03-24 01:55:59,265 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/5d94dcee70d147819e26df86cfa19aee as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee 2023-03-24 01:55:59,268 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee, entries=150, sequenceid=179, filesize=11.8 K 2023-03-24 01:55:59,269 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/6ae80da2b7e04f7abf62063c36f6e10b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b 2023-03-24 01:55:59,271 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b, entries=150, sequenceid=179, filesize=11.8 K 2023-03-24 01:55:59,272 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7db42890c5a058f4b49bb0a2fb398589 in 455ms, sequenceid=179, compaction requested=true 2023-03-24 01:55:59,272 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:55:59,272 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e60a2b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,275 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,282 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,283 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,285 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,285 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,285 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,286 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,286 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,287 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,291 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,291 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,291 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,291 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:55:59,359 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:55:59,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:55:59,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,391 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:59,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:59,397 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:59,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:59,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:59,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,403 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,403 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,403 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:59,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:59,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,411 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:59,412 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,412 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,412 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,413 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,413 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2220bc07[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,415 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,416 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,416 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,417 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,420 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,422 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,422 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,423 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,423 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,425 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,426 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,427 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,429 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,429 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,430 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,430 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623019473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623019473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,531 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:59,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:59,534 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,542 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,542 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,543 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:59,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:59,544 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,544 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:59,544 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,544 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,547 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c874c22[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,550 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,561 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,561 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,561 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,562 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,562 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,563 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,567 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:59,568 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,568 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,669 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:59,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:59,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:59,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:59,672 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:59,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:59,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:59,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,673 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,674 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:59,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:59,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:59,676 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,676 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:59,676 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,676 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,676 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3524735b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,683 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,684 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,684 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,684 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,685 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,685 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,688 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,688 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,691 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:55:59,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,697 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,698 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,697 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,698 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,698 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:59,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/1f8b7f185d194879b0a5590b79f77d2f 2023-03-24 01:55:59,798 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:59,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:59,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:59,801 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:59,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:59,801 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:59,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,803 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:59,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:59,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:59,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:59,806 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,806 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:59,806 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,806 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,806 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25039086[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,810 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,811 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,815 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,816 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,817 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,823 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,824 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:59,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,824 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,824 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,924 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:55:59,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:55:59,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:55:59,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:55:59,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:55:59,927 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:55:59,927 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,927 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:55:59,927 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:55:59,928 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,928 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,929 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:55:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:55:59,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:55:59,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:55:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:55:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:55:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:55:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:55:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:55:59,930 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:55:59,931 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:55:59,931 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:55:59,938 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:55:59,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e4c0a5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:55:59,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:55:59,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:55:59,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:55:59,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:55:59,941 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,941 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:55:59,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:55:59,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:55:59,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:55:59,945 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:55:59,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:55:59,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:55:59,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:55:59,945 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:55:59,947 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,947 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:55:59,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,947 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:55:59,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,949 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:55:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:55:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,951 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:55:59,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:55:59,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:55:59,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:55:59,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:55:59,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,952 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:55:59,952 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:55:59,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:55:59,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:55:59,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623019996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623019996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623019997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,052 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,055 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,056 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,057 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,058 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,059 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,059 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,059 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:00,059 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:00,059 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,061 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,061 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14146a87[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:00,062 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:00,062 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,064 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,064 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,064 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:00,067 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:00,069 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:00,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:00,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:00,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:00,069 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:00,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,071 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,072 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:00,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:00,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:00,075 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:00,075 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:00,075 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,075 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,175 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/7aa0ba12f9094876a0a0fd514b57e259 2023-03-24 01:56:00,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,180 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,180 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,181 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,182 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:00,188 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:00,188 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:00,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc6df5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:00,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:00,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:00,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:00,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:00,199 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:00,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:00,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:00,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:00,199 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:00,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:00,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:00,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:00,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,207 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:00,210 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:00,310 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,313 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,315 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,315 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,315 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,315 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,320 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,324 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,324 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,324 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:00,324 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:00,324 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,326 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b825a84[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:00,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:00,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,328 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:00,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:00,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:00,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:00,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:00,334 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:00,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:00,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:00,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:00,334 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:00,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,336 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:00,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,339 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,340 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:00,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,340 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:00,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,440 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,445 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,447 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,448 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,455 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,455 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,455 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:00,455 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:00,455 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc754b9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:00,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:00,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,461 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:00,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,461 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:00,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:00,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:00,466 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:00,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:00,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:00,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:00,467 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:00,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,473 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:00,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:00,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,478 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:00,478 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:00,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623020509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623020510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623020511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,578 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,583 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,583 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,584 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,585 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,585 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,586 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:00,587 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:00,587 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:00,588 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b05594[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:00,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:00,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:00,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:00,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:00,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:00,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:00,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/dae9dcc974dc4d71a59272328f751e04 2023-03-24 01:56:00,596 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:00,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:00,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:00,599 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:00,599 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,602 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:00,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/1f8b7f185d194879b0a5590b79f77d2f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f 2023-03-24 01:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:00,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:00,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f, entries=200, sequenceid=198, filesize=14.1 K 2023-03-24 01:56:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/7aa0ba12f9094876a0a0fd514b57e259 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:00,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:00,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,610 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:00,610 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:00,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:00,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:00,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 01:56:00,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/dae9dcc974dc4d71a59272328f751e04 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04 2023-03-24 01:56:00,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 01:56:00,620 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7db42890c5a058f4b49bb0a2fb398589 in 1261ms, sequenceid=198, compaction requested=true 2023-03-24 01:56:00,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:00,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:00,620 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:00,620 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:00,620 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:00,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:00,624 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50740 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:00,624 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:00,624 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:00,625 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:00,625 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,624 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,625 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/c18be2c5e3b54e0a957ff531235b2c6c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=47.2 K 2023-03-24 01:56:00,625 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/21e3af9d26854ab88bc2b9acf5f03b72, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=49.6 K 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c18be2c5e3b54e0a957ff531235b2c6c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 21e3af9d26854ab88bc2b9acf5f03b72, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 61ecc4d003924826862322785fa6cefa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679622957331 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ec78bd9b10a04f8da051d11eab51c624, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679622957331 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5d94dcee70d147819e26df86cfa19aee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679622957884 2023-03-24 01:56:00,627 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 34828f41196747e996f5ed4e0b45f884, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679622957884 2023-03-24 01:56:00,628 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7aa0ba12f9094876a0a0fd514b57e259, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:00,628 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1f8b7f185d194879b0a5590b79f77d2f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:00,642 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#426 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:00,643 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:00,665 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/c5d7ead7b88246dd8f06f4be92e4b028 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c5d7ead7b88246dd8f06f4be92e4b028 2023-03-24 01:56:00,670 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into c5d7ead7b88246dd8f06f4be92e4b028(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:00,670 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:00,670 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=12, startTime=1679622960620; duration=0sec 2023-03-24 01:56:00,670 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:00,671 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:00,671 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:00,671 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:00,672 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,672 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/c303b96cd2bf45bab4bd3f3f44744e26, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=47.2 K 2023-03-24 01:56:00,672 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c303b96cd2bf45bab4bd3f3f44744e26, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1679622957191 2023-03-24 01:56:00,672 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9bef35ad07ef460781cb8d81e42e7d03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679622957331 2023-03-24 01:56:00,673 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6ae80da2b7e04f7abf62063c36f6e10b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1679622957884 2023-03-24 01:56:00,673 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting dae9dcc974dc4d71a59272328f751e04, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:00,678 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#428 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:00,700 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/9574fd792f5742aa916b021db5a5df2a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9574fd792f5742aa916b021db5a5df2a 2023-03-24 01:56:00,706 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into 9574fd792f5742aa916b021db5a5df2a(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:00,706 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:00,706 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=12, startTime=1679622960621; duration=0sec 2023-03-24 01:56:00,706 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:00,710 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:00,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:00,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:00,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:00,719 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,720 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:00,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:00,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:00,721 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:00,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:00,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:00,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:00,722 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:00,722 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:00,722 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 01:56:00,722 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:00,722 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:00,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:00,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:00,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:00,723 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:01,069 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/526114cb8eff4feb981c8b3702e898cb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/526114cb8eff4feb981c8b3702e898cb 2023-03-24 01:56:01,073 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 526114cb8eff4feb981c8b3702e898cb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:01,073 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:01,073 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=12, startTime=1679622960620; duration=0sec 2023-03-24 01:56:01,074 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:01,149 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/40065a74d27a41c48f012be66b0a3041 2023-03-24 01:56:01,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:01,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:01,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623021529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623021530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623021533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623021533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623021534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,561 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/0a2092e8fbfb46fe98cfbc1b975b7320 2023-03-24 01:56:01,577 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/ed7c7978b759412f92bbd317cf209d8b 2023-03-24 01:56:01,581 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/40065a74d27a41c48f012be66b0a3041 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041 2023-03-24 01:56:01,585 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041, entries=150, sequenceid=213, filesize=11.8 K 2023-03-24 01:56:01,585 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/0a2092e8fbfb46fe98cfbc1b975b7320 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320 2023-03-24 01:56:01,588 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320, entries=150, sequenceid=213, filesize=11.8 K 2023-03-24 01:56:01,589 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/ed7c7978b759412f92bbd317cf209d8b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b 2023-03-24 01:56:01,593 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b, entries=150, sequenceid=213, filesize=11.8 K 2023-03-24 01:56:01,594 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7db42890c5a058f4b49bb0a2fb398589 in 871ms, sequenceid=213, compaction requested=false 2023-03-24 01:56:01,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:01,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:01,597 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,597 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:01,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490256b2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:01,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:01,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:01,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:01,604 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:01,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:01,607 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:01,607 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:01,607 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:01,607 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:01,607 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:01,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,613 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:01,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,632 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:01,632 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:01,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:01,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=114.05 KB heapSize=340.13 KB 2023-03-24 01:56:01,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1679622961531, max timestamp=1679622961650Num uniques 859; 2023-03-24 01:56:01,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=8046, suffix heap size=117064,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=176 2023-03-24 01:56:01,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679622961531, max timestamp=1679622961650Num uniques 859; 2023-03-24 01:56:01,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1679622961531, max timestamp=1679622961650Num uniques 859; 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:01,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:01,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:01,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:01,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/33ca2e3e89e74d67a3dbc74426519909 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622961689, max timestamp=1679622961704Num uniques 859; 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622961689, max timestamp=1679622961704Num uniques 859; 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622961689, max timestamp=1679622961704Num uniques 859; 2023-03-24 01:56:01,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:01,732 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:01,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:01,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:01,746 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:01,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:01,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:01,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:01,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,747 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,749 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:01,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:01,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:01,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623021743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623021744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623021745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:01,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:01,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623021753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:01,788 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:01,788 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:01,790 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,790 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:01,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607d636c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:01,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:01,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:01,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:01,799 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:01,802 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:01,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:01,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:01,802 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:01,802 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:01,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,804 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,804 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,806 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:01,808 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:01,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:01,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:01,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,809 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:01,809 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,809 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623021855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623021856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623021856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:01,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623021866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,909 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:01,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:01,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:01,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:01,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:01,915 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:01,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:01,915 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:01,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:01,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:01,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:01,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:01,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:01,922 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:01,922 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:01,922 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:01,922 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:01,922 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:01,923 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:01,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ba8146[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:01,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:01,929 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,930 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:01,930 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:01,930 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,931 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:01,932 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:01,934 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:01,934 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:01,934 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:01,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:01,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:01,948 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:01,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:01,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:01,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:01,953 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:01,954 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:01,954 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:01,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:02,054 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:02,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:02,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:02,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:02,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:02,065 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:02,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:02,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,067 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,067 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,068 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:02,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623022059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623022060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623022061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623022073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:02,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8c6090[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:02,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:02,093 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:02,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:02,100 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,101 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:02,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:02,103 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:02,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:02,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:02,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:02,103 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:02,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/1908fa44450c4fe890d6235b2a5ba39e 2023-03-24 01:56:02,105 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,105 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,105 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,106 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,108 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,119 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,119 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,119 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:02,120 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:02,220 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:02,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:02,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:02,223 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:02,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:02,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:02,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,224 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:02,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:02,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:02,227 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:02,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:02,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:02,228 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:02,228 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:02,230 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e01c15[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:02,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:02,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,232 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:02,236 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:02,236 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,236 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:02,236 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:02,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:02,239 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:02,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:02,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:02,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:02,239 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:02,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,240 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,241 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:02,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:02,245 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:02,245 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:02,245 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:02,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,345 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:02,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:02,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:02,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:02,349 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:02,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:02,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:02,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:02,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:02,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:02,358 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,358 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:02,358 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:02,358 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:02,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:02,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:02,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:02,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f8d6fe6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:02,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:02,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,367 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:02,367 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:02,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:02,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:02,369 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:02,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:02,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:02,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:02,369 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:02,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,372 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:02,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:02,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,376 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:02,377 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:02,376 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,377 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,377 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,377 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:02,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623022376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623022377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623022377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623022385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,478 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:02,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:02,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:02,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:02,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:02,492 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:02,492 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,492 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:02,492 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:02,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,495 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:02,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:02,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:02,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,503 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:02,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:02,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:02,504 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:02,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:02,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,507 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,508 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,508 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:02,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eac79e0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:02,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:02,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,510 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:02,510 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:02,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:02,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:02,513 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:02,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:02,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:02,514 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:02,514 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:02,515 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:02,515 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,515 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:02,517 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,517 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,517 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:02,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,519 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,520 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:02,526 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,526 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:02,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:02,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.95 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/eae6877f863d4170a2c0833442a128d2 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:02,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:02,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:02,527 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,527 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:02,527 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:02,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:02,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/33ca2e3e89e74d67a3dbc74426519909 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909 2023-03-24 01:56:02,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909, entries=200, sequenceid=236, filesize=14.1 K 2023-03-24 01:56:02,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/1908fa44450c4fe890d6235b2a5ba39e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e 2023-03-24 01:56:02,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e, entries=150, sequenceid=236, filesize=11.8 K 2023-03-24 01:56:02,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/eae6877f863d4170a2c0833442a128d2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2 2023-03-24 01:56:02,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2, entries=150, sequenceid=236, filesize=11.8 K 2023-03-24 01:56:02,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~38.78 KB/39712, heapSize ~97.99 KB/100344, currentSize=154.31 KB/158010 for 7db42890c5a058f4b49bb0a2fb398589 in 902ms, sequenceid=236, compaction requested=true 2023-03-24 01:56:02,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:02,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:02,553 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:02,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:02,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:02,553 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:02,555 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:02,555 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:02,555 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,555 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c5d7ead7b88246dd8f06f4be92e4b028, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=37.9 K 2023-03-24 01:56:02,555 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:02,555 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:02,555 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,555 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/526114cb8eff4feb981c8b3702e898cb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=35.6 K 2023-03-24 01:56:02,555 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c5d7ead7b88246dd8f06f4be92e4b028, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:02,556 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 526114cb8eff4feb981c8b3702e898cb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:02,556 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 40065a74d27a41c48f012be66b0a3041, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679622959369 2023-03-24 01:56:02,556 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 0a2092e8fbfb46fe98cfbc1b975b7320, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679622959369 2023-03-24 01:56:02,556 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 33ca2e3e89e74d67a3dbc74426519909, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:02,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1908fa44450c4fe890d6235b2a5ba39e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:02,590 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#435 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:02,601 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:02,613 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/39c52d2c036a4ad79eb95523ba50faba as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39c52d2c036a4ad79eb95523ba50faba 2023-03-24 01:56:02,617 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 39c52d2c036a4ad79eb95523ba50faba(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:02,617 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:02,617 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622962553; duration=0sec 2023-03-24 01:56:02,617 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:02,617 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:02,618 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:02,618 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:02,618 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,618 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9574fd792f5742aa916b021db5a5df2a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=35.6 K 2023-03-24 01:56:02,618 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9574fd792f5742aa916b021db5a5df2a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679622959034 2023-03-24 01:56:02,618 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ed7c7978b759412f92bbd317cf209d8b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679622959369 2023-03-24 01:56:02,619 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting eae6877f863d4170a2c0833442a128d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:02,628 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:02,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:02,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:02,641 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:02,648 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/53816ff13f0546249a38b39b12863d11 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/53816ff13f0546249a38b39b12863d11 2023-03-24 01:56:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:02,652 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:02,652 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,652 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:02,652 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:02,652 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,653 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,655 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:02,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:02,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:02,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:02,657 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 53816ff13f0546249a38b39b12863d11(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:02,657 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:02,657 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622962553; duration=0sec 2023-03-24 01:56:02,657 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:02,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:02,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:02,660 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:02,660 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:02,660 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:02,661 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:02,685 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/561e60e02fbd45878070a4b5b4d15605 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/561e60e02fbd45878070a4b5b4d15605 2023-03-24 01:56:02,702 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into 561e60e02fbd45878070a4b5b4d15605(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:02,702 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:02,702 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622962553; duration=0sec 2023-03-24 01:56:02,702 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:02,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:02,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:02,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623022899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623022903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623022903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:02,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:02,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623022904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623023005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623023013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623023013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623023017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,118 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/e9cc5db549624d4e8620ae265e9a07bb 2023-03-24 01:56:03,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623023214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623023216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623023217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623023225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35054 deadline: 1679623023526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35020 deadline: 1679623023528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623023528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623023532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:03,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35074 deadline: 1679623023541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,554 DEBUG [Thread-3092] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:03,559 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/29a075c10e5943da959b1a1135b4aac9 2023-03-24 01:56:03,588 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/ecef18fc03fb4134851cee3f13bb6e78 2023-03-24 01:56:03,604 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/e9cc5db549624d4e8620ae265e9a07bb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb 2023-03-24 01:56:03,609 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb, entries=150, sequenceid=275, filesize=11.9 K 2023-03-24 01:56:03,610 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/29a075c10e5943da959b1a1135b4aac9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9 2023-03-24 01:56:03,637 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9, entries=150, sequenceid=275, filesize=11.9 K 2023-03-24 01:56:03,638 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/ecef18fc03fb4134851cee3f13bb6e78 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78 2023-03-24 01:56:03,648 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78, entries=150, sequenceid=275, filesize=11.9 K 2023-03-24 01:56:03,649 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=40.25 KB/41220 for 7db42890c5a058f4b49bb0a2fb398589 in 989ms, sequenceid=275, compaction requested=false 2023-03-24 01:56:03,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:03,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:03,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:03,654 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:03,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22281445[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:03,654 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:03,657 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,658 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:03,660 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:03,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:03,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:03,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,661 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:03,663 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:03,663 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:03,670 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:03,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:03,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:03,670 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:03,670 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:03,673 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:03,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:03,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:03,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,676 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:03,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:03,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:03,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:03,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:03,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:03,679 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:03,678 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:03,679 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:03,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:03,779 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:03,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:03,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:03,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:03,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:03,792 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:03,792 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:03,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,793 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:03,794 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:03,794 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:03,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:03,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:03,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:03,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:03,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:03,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:03,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:03,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:03,808 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:03,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:03,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:03,821 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:03,821 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:04,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622964061, max timestamp=1679622964085Num uniques 859; 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622964061, max timestamp=1679622964085Num uniques 859; 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622964061, max timestamp=1679622964085Num uniques 859; 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:04,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:04,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622964117, max timestamp=1679622964133Num uniques 1018; 2023-03-24 01:56:04,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622964117, max timestamp=1679622964133Num uniques 1018; 2023-03-24 01:56:04,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:56:04,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622964117, max timestamp=1679622964133Num uniques 1018; 2023-03-24 01:56:04,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:56:04,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 01:56:04,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623024167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623024174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623024175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623024175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,262 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/529336c141c646518941ed7b13176602 2023-03-24 01:56:04,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623024278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623024287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623024291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623024291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623024491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623024499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623024504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35048 deadline: 1679623024504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,680 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/9a250d89cc8e4b0c98b198b4d411174e 2023-03-24 01:56:04,696 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/2c1707b74a0348ec8ef9e6fb3c991896 2023-03-24 01:56:04,701 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/529336c141c646518941ed7b13176602 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602 2023-03-24 01:56:04,705 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602, entries=150, sequenceid=285, filesize=11.9 K 2023-03-24 01:56:04,706 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/9a250d89cc8e4b0c98b198b4d411174e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e 2023-03-24 01:56:04,717 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e, entries=150, sequenceid=285, filesize=11.9 K 2023-03-24 01:56:04,718 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/2c1707b74a0348ec8ef9e6fb3c991896 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896 2023-03-24 01:56:04,729 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896, entries=150, sequenceid=285, filesize=11.9 K 2023-03-24 01:56:04,730 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 7db42890c5a058f4b49bb0a2fb398589 in 909ms, sequenceid=285, compaction requested=true 2023-03-24 01:56:04,730 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:04,730 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:04,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:04,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:04,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:04,733 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169ca4a1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:04,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:04,735 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:04,740 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,740 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:04,742 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:04,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:04,758 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:04,759 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:04,759 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:04,759 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:04,759 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:04,763 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,763 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:04,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,763 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,765 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:04,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:04,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:04,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:04,769 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:04,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:04,769 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:04,770 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:04,770 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:04,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:04,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:04,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/b1ceb8bffeb442d7b0e6e46891af23aa 2023-03-24 01:56:04,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623024825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623024826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623024827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623024828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,870 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:04,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:04,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:04,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:04,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:04,874 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:04,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:04,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:04,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:04,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:04,878 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:04,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:04,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:04,884 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:04,884 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:04,884 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:04,884 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:04,885 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:04,885 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:04,887 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,888 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ba1bcb[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:04,891 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:04,891 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,893 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,893 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:04,894 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:04,894 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:04,897 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:04,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:04,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:04,897 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:04,897 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:04,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,900 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:04,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,901 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:04,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:04,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:04,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:04,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:04,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:04,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:04,906 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:04,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:04,907 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:04,907 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:04,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623024942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623024942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623024942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:04,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:04,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623024943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,007 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,019 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,038 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,038 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,038 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:05,038 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:05,045 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,045 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,047 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:05,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f940905[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:05,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:05,048 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,054 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:05,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:05,054 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:05,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,055 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:05,056 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:05,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:05,060 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:05,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:05,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:05,061 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:05,061 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:05,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,063 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,070 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:05,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:05,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,071 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:05,071 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:05,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623025152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623025153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623025154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623025154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,171 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,175 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,175 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,177 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,179 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,180 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,180 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,180 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:05,180 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:05,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:05,182 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,182 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:05,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d7cbe8e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:05,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:05,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:05,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,187 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:05,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:05,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:05,190 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:05,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:05,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:05,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:05,191 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:05,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,192 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,194 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,198 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:05,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,198 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:05,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,198 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:05,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/a5528378ec3646b4956290cfd50beefe 2023-03-24 01:56:05,298 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,302 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,302 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,303 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,305 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,305 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,306 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:05,307 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:05,308 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,308 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,309 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:05,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6a62c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:05,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:05,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,311 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:05,312 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:05,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:05,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,313 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:05,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:05,317 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:05,317 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:05,317 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:05,317 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:05,317 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:05,320 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,320 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,320 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,321 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,322 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:05,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:05,325 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:05,325 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:05,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:05,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,425 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,429 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,430 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,430 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,431 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,431 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,432 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,432 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,432 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:05,432 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:05,434 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,435 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:05,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c408272[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:05,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:05,436 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,439 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,439 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:05,440 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,440 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:05,444 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:05,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:05,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:05,444 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:05,444 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:05,446 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,446 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,447 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,447 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,450 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,450 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:05,451 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:05,451 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:05,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623025462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623025462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623025463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:05,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623025464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,551 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,554 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,554 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,555 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,556 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,556 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,557 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,557 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:05,559 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:05,561 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,561 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23336ede[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:05,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:05,563 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,564 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:05,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,564 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:05,565 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:05,568 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:05,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:05,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:05,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:05,568 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:05,572 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,572 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,572 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:05,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,572 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:05,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,574 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:05,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:05,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:05,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:05,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:05,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:05,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:05,583 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:05,583 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:05,582 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:05,583 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/bdec12ab933c40d1bc6e4b3328e6d172 2023-03-24 01:56:05,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/b1ceb8bffeb442d7b0e6e46891af23aa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa 2023-03-24 01:56:05,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa, entries=200, sequenceid=339, filesize=14.3 K 2023-03-24 01:56:05,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/a5528378ec3646b4956290cfd50beefe as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe 2023-03-24 01:56:05,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe, entries=150, sequenceid=339, filesize=11.9 K 2023-03-24 01:56:05,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/bdec12ab933c40d1bc6e4b3328e6d172 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172 2023-03-24 01:56:05,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172, entries=150, sequenceid=339, filesize=11.9 K 2023-03-24 01:56:05,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 7db42890c5a058f4b49bb0a2fb398589 in 857ms, sequenceid=339, compaction requested=true 2023-03-24 01:56:05,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:05,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:05,669 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:05,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:05,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:05,670 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:05,672 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51478 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:05,672 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:05,672 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:05,672 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,673 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:05,673 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39c52d2c036a4ad79eb95523ba50faba, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=50.3 K 2023-03-24 01:56:05,673 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,673 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/53816ff13f0546249a38b39b12863d11, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=47.9 K 2023-03-24 01:56:05,673 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 39c52d2c036a4ad79eb95523ba50faba, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:05,673 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 53816ff13f0546249a38b39b12863d11, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:05,673 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e9cc5db549624d4e8620ae265e9a07bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679622961724 2023-03-24 01:56:05,673 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 29a075c10e5943da959b1a1135b4aac9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679622961724 2023-03-24 01:56:05,674 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 529336c141c646518941ed7b13176602, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679622962901 2023-03-24 01:56:05,674 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9a250d89cc8e4b0c98b198b4d411174e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679622962901 2023-03-24 01:56:05,674 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b1ceb8bffeb442d7b0e6e46891af23aa, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:05,674 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a5528378ec3646b4956290cfd50beefe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:05,683 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:05,683 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#447 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:05,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:05,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:05,685 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:05,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:05,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:05,688 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:05,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:05,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:05,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,688 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:05,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:05,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:05,690 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:05,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:05,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:05,704 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:05,704 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:05,704 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 01:56:05,707 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:05,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:05,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:05,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:05,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:05,708 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:05,710 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e5a846e7fd284a289fd1fbbbc8048fb8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e5a846e7fd284a289fd1fbbbc8048fb8 2023-03-24 01:56:05,714 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into e5a846e7fd284a289fd1fbbbc8048fb8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:05,714 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:05,714 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=12, startTime=1679622965669; duration=0sec 2023-03-24 01:56:05,715 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:05,715 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:05,715 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:05,716 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:05,716 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:05,716 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/561e60e02fbd45878070a4b5b4d15605, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=47.9 K 2023-03-24 01:56:05,716 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 561e60e02fbd45878070a4b5b4d15605, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679622961531 2023-03-24 01:56:05,717 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ecef18fc03fb4134851cee3f13bb6e78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679622961724 2023-03-24 01:56:05,717 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2c1707b74a0348ec8ef9e6fb3c991896, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679622962901 2023-03-24 01:56:05,718 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting bdec12ab933c40d1bc6e4b3328e6d172, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:05,718 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/4711b84b025a4fcd9f86ce08f17fd12a 2023-03-24 01:56:05,725 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:05,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:05,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:06,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622965997, max timestamp=1679622966013Num uniques 859; 2023-03-24 01:56:06,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:06,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622965997, max timestamp=1679622966013Num uniques 859; 2023-03-24 01:56:06,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:06,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622965997, max timestamp=1679622966013Num uniques 859; 2023-03-24 01:56:06,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:06,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623026051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622966032, max timestamp=1679622966065Num uniques 1068; 2023-03-24 01:56:06,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 01:56:06,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622966032, max timestamp=1679622966065Num uniques 1068; 2023-03-24 01:56:06,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 01:56:06,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679622966032, max timestamp=1679622966065Num uniques 1068; 2023-03-24 01:56:06,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 01:56:06,097 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/03517cda1e3c4ad79d53359913493e7b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/03517cda1e3c4ad79d53359913493e7b 2023-03-24 01:56:06,102 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 03517cda1e3c4ad79d53359913493e7b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:06,103 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:06,103 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=12, startTime=1679622965669; duration=0sec 2023-03-24 01:56:06,103 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:06,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623026115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623026116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623026116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,137 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/479b0019814b4d51815e1588555d8099 2023-03-24 01:56:06,148 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/e250938696ef42f6b6c2d9faf72e421d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e250938696ef42f6b6c2d9faf72e421d 2023-03-24 01:56:06,152 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into e250938696ef42f6b6c2d9faf72e421d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:06,152 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:06,153 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=12, startTime=1679622965669; duration=0sec 2023-03-24 01:56:06,153 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:06,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35036 deadline: 1679623026168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623026220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623026227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623026227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623026374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623026425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623026434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623026434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,546 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/53b33ad535ce485991fd411785a17f0c 2023-03-24 01:56:06,550 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/4711b84b025a4fcd9f86ce08f17fd12a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a 2023-03-24 01:56:06,554 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 01:56:06,554 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/479b0019814b4d51815e1588555d8099 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099 2023-03-24 01:56:06,558 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 01:56:06,558 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/53b33ad535ce485991fd411785a17f0c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c 2023-03-24 01:56:06,562 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 01:56:06,562 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 7db42890c5a058f4b49bb0a2fb398589 in 858ms, sequenceid=347, compaction requested=false 2023-03-24 01:56:06,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:06,563 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:06,564 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b332cf8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:06,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:06,566 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,568 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:06,568 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:06,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,568 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:06,569 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,569 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:06,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:06,573 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:06,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:06,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:06,573 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:06,573 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:06,574 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,574 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,574 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,575 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,576 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:06,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:06,578 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,579 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:06,579 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:06,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:06,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,679 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:06,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:06,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:06,682 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:06,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:06,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:06,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,682 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,684 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,684 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,685 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:06,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:06,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:06,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:06,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:06,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,686 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:06,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622966113, max timestamp=1679622966685Num uniques 1127; 2023-03-24 01:56:06,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 01:56:06,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622966113, max timestamp=1679622966685Num uniques 1127; 2023-03-24 01:56:06,687 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 01:56:06,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:06,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=69.33 KB heapSize=188.11 KB 2023-03-24 01:56:06,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:06,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:06,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:06,688 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:06,688 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:06,688 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:06,688 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:06,688 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:06,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:06,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:06,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 01:56:06,690 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,690 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:06,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7874[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:06,692 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:06,692 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,693 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:06,693 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:06,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,693 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:06,694 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,694 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:06,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:06,698 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:06,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:06,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:06,698 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:06,698 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:06,703 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,703 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,708 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,708 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,709 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,713 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:06,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,713 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,713 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:06,713 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:06,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:06,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623026755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623026761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623026762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623026763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,813 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:06,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:06,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:06,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:06,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:06,817 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:06,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:06,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:06,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,817 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,818 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,819 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:06,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:06,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:06,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:06,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:06,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:06,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:06,820 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:06,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:06,821 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,822 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:06,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8c43917[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:06,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:06,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:06,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:06,827 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:06,827 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:06,828 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:06,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,828 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:06,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:06,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:06,832 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:06,832 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:06,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,835 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,837 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:06,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:06,839 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,840 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,840 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:06,840 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:06,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623026864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623026870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623026873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:06,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623026873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,940 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:06,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:06,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:06,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:06,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:06,943 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:06,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:06,943 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:06,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,951 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,951 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:06,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:06,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:06,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:06,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:06,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:06,953 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:06,953 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:06,954 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:06,955 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,955 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,956 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:06,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3039f34f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:06,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:06,957 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:06,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:06,959 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,961 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:06,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:06,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:06,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:06,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,964 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:06,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,964 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:06,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,966 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:06,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:06,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,969 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:06,969 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:06,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:06,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:06,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:06,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:06,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:06,970 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:06,970 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:06,970 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:06,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:06,970 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,070 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:07,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:07,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:07,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:07,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:07,073 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:07,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:07,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:07,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,075 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:07,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:07,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623027071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:07,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:07,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:07,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:07,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:07,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:07,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:07,078 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:07,078 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:07,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:07,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:07,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:07,079 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,080 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:07,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623027074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:07,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f4d5c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:07,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:07,083 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623027079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:07,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:07,085 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:07,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:07,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,086 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:07,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:07,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:07,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:07,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:07,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:07,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:07,088 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:07,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,090 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,090 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623027082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,091 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:07,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:07,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,097 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:07,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:07,098 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:07,097 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f8e5e4a1dd764bb2abdda7377bb62a9a 2023-03-24 01:56:07,198 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:07,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:07,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:07,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:07,201 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:07,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:07,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:07,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,203 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:07,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:07,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:07,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:07,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:07,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:07,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:07,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:07,204 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,204 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:07,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:07,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:07,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,206 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:07,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:07,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797849bb[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:07,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:07,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:07,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:07,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:07,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,209 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:07,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,211 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:07,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:07,214 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:07,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:07,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:07,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:07,214 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:07,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:07,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:07,222 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,222 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:07,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,222 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:07,323 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:07,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:07,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:07,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:07,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:07,326 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:07,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:07,326 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:07,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,328 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,328 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:07,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:07,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:07,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:07,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:07,330 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,330 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:07,330 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:07,330 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:07,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:07,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,333 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:07,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dddb02[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:07,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:07,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,335 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,336 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:07,336 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:07,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:07,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:07,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:07,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:07,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:07,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:07,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:07,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,344 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:07,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:07,349 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:07,349 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:07,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:07,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623027378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623027383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623027386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623027391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,449 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:07,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:07,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:07,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:07,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:07,453 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:07,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:07,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:07,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,453 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,454 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,454 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:07,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:07,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:07,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:07,456 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:07,456 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,457 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:07,457 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:07,457 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:07,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:07,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:07,457 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:07,458 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,458 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,459 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:07,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f81f9dd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:07,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:07,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,461 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:07,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,464 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:07,464 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:07,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:07,464 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:07,466 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,466 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,466 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:07,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:07,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:07,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:07,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:07,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:07,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:07,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:07,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:07,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:07,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:07,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:07,473 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,473 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:07,473 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:07,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e25d93cd813b45a3ba718bdb32a8c87b 2023-03-24 01:56:07,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/73d5dd1a616745fbb2aab01785b5c793 2023-03-24 01:56:07,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/f8e5e4a1dd764bb2abdda7377bb62a9a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a 2023-03-24 01:56:07,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a, entries=200, sequenceid=405, filesize=14.3 K 2023-03-24 01:56:07,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e25d93cd813b45a3ba718bdb32a8c87b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b 2023-03-24 01:56:07,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b, entries=150, sequenceid=405, filesize=11.9 K 2023-03-24 01:56:07,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/73d5dd1a616745fbb2aab01785b5c793 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793 2023-03-24 01:56:07,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793, entries=150, sequenceid=405, filesize=11.9 K 2023-03-24 01:56:07,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~188.06 KB/192576, currentSize=114.05 KB/116790 for 7db42890c5a058f4b49bb0a2fb398589 in 875ms, sequenceid=405, compaction requested=true 2023-03-24 01:56:07,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:07,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:07,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:07,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 01:56:07,562 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:07,562 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:07,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:07,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:07,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:07,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:07,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,567 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e250938696ef42f6b6c2d9faf72e421d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.3 K 2023-03-24 01:56:07,567 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/03517cda1e3c4ad79d53359913493e7b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=38.7 K 2023-03-24 01:56:07,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e250938696ef42f6b6c2d9faf72e421d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:07,568 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 03517cda1e3c4ad79d53359913493e7b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:07,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 53b33ad535ce485991fd411785a17f0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679622964814 2023-03-24 01:56:07,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 73d5dd1a616745fbb2aab01785b5c793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:07,568 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4711b84b025a4fcd9f86ce08f17fd12a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679622964814 2023-03-24 01:56:07,569 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting f8e5e4a1dd764bb2abdda7377bb62a9a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:07,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622966760, max timestamp=1679622967571Num uniques 859; 2023-03-24 01:56:07,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,573 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:07,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622966760, max timestamp=1679622967571Num uniques 859; 2023-03-24 01:56:07,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622966760, max timestamp=1679622967571Num uniques 859; 2023-03-24 01:56:07,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:07,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:07,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:07,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:07,578 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,578 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,581 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:07,581 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:07,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:07,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,582 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:07,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:07,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:07,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:07,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:07,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:07,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:07,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:07,588 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:07,589 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:07,591 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:07,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:07,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:07,593 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=46.96 KB heapSize=126 KB 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:07,593 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:07,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:07,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:07,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:07,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:07,639 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/e98a3a8ae7334353a0b9e2772351aee4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e98a3a8ae7334353a0b9e2772351aee4 2023-03-24 01:56:07,644 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into e98a3a8ae7334353a0b9e2772351aee4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:07,645 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:07,645 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622967562; duration=0sec 2023-03-24 01:56:07,645 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:07,645 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:07,646 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:07,646 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:07,646 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:07,646 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e5a846e7fd284a289fd1fbbbc8048fb8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.3 K 2023-03-24 01:56:07,646 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e5a846e7fd284a289fd1fbbbc8048fb8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1679622964148 2023-03-24 01:56:07,646 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 479b0019814b4d51815e1588555d8099, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679622964814 2023-03-24 01:56:07,647 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e25d93cd813b45a3ba718bdb32a8c87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:07,653 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#459 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622967683, max timestamp=1679622967711Num uniques 859; 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622967683, max timestamp=1679622967711Num uniques 859; 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622967683, max timestamp=1679622967711Num uniques 859; 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622967729, max timestamp=1679622967807Num uniques 1018; 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622967729, max timestamp=1679622967807Num uniques 1018; 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679622967729, max timestamp=1679622967807Num uniques 1018; 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 01:56:07,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 01:56:07,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623027896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623027897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623027899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623027903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:07,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:07,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623027904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623028005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623028006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623028012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,029 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/be381b6bd2a34211b313fa013884150b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/be381b6bd2a34211b313fa013884150b 2023-03-24 01:56:08,039 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into be381b6bd2a34211b313fa013884150b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:08,039 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:08,039 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622967562; duration=0sec 2023-03-24 01:56:08,039 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:08,051 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/8828cb2b4ff145b4b5de30331eff220b 2023-03-24 01:56:08,065 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/228caa53734a4eb2a2410ff246923497 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/228caa53734a4eb2a2410ff246923497 2023-03-24 01:56:08,069 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 228caa53734a4eb2a2410ff246923497(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:08,069 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:08,069 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622967562; duration=0sec 2023-03-24 01:56:08,069 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:08,071 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e33120abf59b43ccbbb6e352ca623cf9 2023-03-24 01:56:08,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623028213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623028214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623028218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,482 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/aa36655eba6440a187ba2162912bd5f7 2023-03-24 01:56:08,485 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/8828cb2b4ff145b4b5de30331eff220b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b 2023-03-24 01:56:08,489 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b, entries=150, sequenceid=429, filesize=11.9 K 2023-03-24 01:56:08,490 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e33120abf59b43ccbbb6e352ca623cf9 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9 2023-03-24 01:56:08,493 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9, entries=150, sequenceid=429, filesize=11.9 K 2023-03-24 01:56:08,493 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/aa36655eba6440a187ba2162912bd5f7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7 2023-03-24 01:56:08,497 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7, entries=150, sequenceid=429, filesize=11.9 K 2023-03-24 01:56:08,497 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~125.95 KB/128976, currentSize=140.89 KB/144270 for 7db42890c5a058f4b49bb0a2fb398589 in 904ms, sequenceid=429, compaction requested=false 2023-03-24 01:56:08,497 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:08,498 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,498 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:08,498 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:08,498 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,498 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:08,498 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:08,499 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:08,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574c74dc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:08,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:08,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:08,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:08,507 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:08,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,507 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,507 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:08,513 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:08,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:08,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:08,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:08,513 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:08,516 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,516 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,516 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,517 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,518 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:08,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,521 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,521 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,521 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,521 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1679622967807, max timestamp=1679622968525Num uniques 1077; 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622967807, max timestamp=1679622968525Num uniques 1077; 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679622967807, max timestamp=1679622968525Num uniques 1077; 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-24 01:56:08,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-24 01:56:08,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:08,535 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=53.67 KB heapSize=144.23 KB 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:08,536 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622968559, max timestamp=1679622968577Num uniques 859; 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622968559, max timestamp=1679622968577Num uniques 859; 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622968559, max timestamp=1679622968577Num uniques 859; 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:08,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:08,621 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:08,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623028608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:08,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623028615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:08,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623028616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:08,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:08,625 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:08,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:08,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:08,626 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,626 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,628 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,628 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:08,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:08,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:08,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:08,630 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,630 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:08,630 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:08,630 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,630 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:08,631 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:08,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:08,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c42da1f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:08,634 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:08,634 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,634 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:08,634 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,639 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:08,640 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:08,640 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,640 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:08,640 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:08,641 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:08,641 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,641 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,641 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:08,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:08,643 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:08,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:08,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:08,643 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:08,643 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:08,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,645 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,645 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,646 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,648 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,648 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,648 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:08,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:08,648 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,648 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,649 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:08,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,649 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:08,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,649 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623028723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623028723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623028725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,749 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:08,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:08,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:08,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:08,752 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:08,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:08,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:08,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,754 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:08,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:08,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:08,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:08,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:08,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:08,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:08,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:08,756 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,756 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:08,756 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:08,757 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,757 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:08,757 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:08,757 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,757 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:08,757 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:08,758 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,759 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,759 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:08,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f9195e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:08,760 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:08,760 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:08,762 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,763 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:08,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:08,763 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:08,764 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:08,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,765 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:08,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:08,767 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:08,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:08,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:08,767 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:08,767 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:08,774 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,774 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,774 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,775 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:08,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:08,778 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:08,778 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,779 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:08,779 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,779 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:08,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,779 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,779 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,879 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:08,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:08,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:08,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:08,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:08,882 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:08,882 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:08,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:08,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:08,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:08,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:08,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:08,886 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,886 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:08,886 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:08,886 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:08,886 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:08,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:08,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:08,887 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:08,891 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,891 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:08,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72165dcd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:08,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:08,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,897 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:08,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:08,898 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:08,898 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,898 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:08,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:08,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:08,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:08,901 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:08,901 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:08,902 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:08,902 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:08,902 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:08,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,903 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:08,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:08,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,907 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:08,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:08,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:08,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:08,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:08,910 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:08,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:08,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:08,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:08,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:08,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:08,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:08,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:08,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:08,911 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:08,911 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:08,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623028911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623028913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623028928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623028929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:08,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623028933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:08,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/edf9a9052e274ffda0fb39ec1d4dbab2 2023-03-24 01:56:09,011 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,015 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,017 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,018 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,022 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c82dc7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,023 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,023 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,027 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,029 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,029 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,034 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,038 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,038 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:09,038 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,038 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,138 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,141 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,141 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,141 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,141 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,142 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,142 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,143 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,145 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,145 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,147 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,147 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c209523[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,149 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,149 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,150 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,150 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,150 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,150 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,151 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,152 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,152 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,154 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,154 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,154 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,154 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,154 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,156 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,156 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,156 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,156 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,164 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,164 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,164 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,164 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,164 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,165 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,165 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623029233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623029234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623029240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,265 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,268 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,269 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,269 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,269 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,269 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,269 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,272 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,272 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,272 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,273 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,273 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,274 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,274 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26094810[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,276 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,276 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,282 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,283 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,283 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,284 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,286 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,286 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,286 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,286 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,287 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:09,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,293 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,293 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,294 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,294 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,357 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/f990738ba49e42978049672f623341e5 2023-03-24 01:56:09,393 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,396 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,400 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,400 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,400 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,400 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,401 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,401 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,401 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,401 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,401 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,403 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66318129[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,405 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,405 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,406 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,406 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,406 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,406 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,406 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,406 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,408 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,408 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,410 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,410 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,410 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,410 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,410 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,413 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,413 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,413 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,413 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,414 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,416 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,416 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,417 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,417 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,417 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,417 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,517 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,520 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,520 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,520 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,520 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,521 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,521 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,522 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,522 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,522 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,522 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,524 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,524 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,524 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,524 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,524 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,525 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,525 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,526 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,526 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61902e3d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,529 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,530 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,531 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,535 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,535 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,535 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,536 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,536 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,536 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,539 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,541 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,541 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,541 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,652 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,655 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,655 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,656 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,658 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,658 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,659 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,659 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,659 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,659 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,660 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,661 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,661 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,662 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592136a1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,663 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,664 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,664 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,664 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,669 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,669 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,672 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,672 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,672 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,673 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,673 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,673 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,674 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,675 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:09,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,678 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,678 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,679 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,679 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,679 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623029743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623029743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:09,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623029751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/7a638180f8154839910b4596588309dc 2023-03-24 01:56:09,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/edf9a9052e274ffda0fb39ec1d4dbab2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2 2023-03-24 01:56:09,779 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2, entries=200, sequenceid=492, filesize=14.3 K 2023-03-24 01:56:09,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/f990738ba49e42978049672f623341e5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5 2023-03-24 01:56:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,782 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,782 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,783 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,783 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,784 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,784 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5, entries=150, sequenceid=492, filesize=11.9 K 2023-03-24 01:56:09,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/7a638180f8154839910b4596588309dc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc 2023-03-24 01:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,786 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:09,787 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,787 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,789 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:09,790 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc, entries=150, sequenceid=492, filesize=11.9 K 2023-03-24 01:56:09,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.11 KB/168048, currentSize=127.47 KB/130530 for 7db42890c5a058f4b49bb0a2fb398589 in 1255ms, sequenceid=492, compaction requested=true 2023-03-24 01:56:09,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:09,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:09,791 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:09,791 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:09,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:09,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,793 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:09,793 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,793 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,794 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:09,794 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/be381b6bd2a34211b313fa013884150b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=38.8 K 2023-03-24 01:56:09,794 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:09,794 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,794 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/228caa53734a4eb2a2410ff246923497, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.4 K 2023-03-24 01:56:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,794 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting be381b6bd2a34211b313fa013884150b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:09,794 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 228caa53734a4eb2a2410ff246923497, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:09,794 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8828cb2b4ff145b4b5de30331eff220b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679622966762 2023-03-24 01:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3504ac5d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:09,795 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e33120abf59b43ccbbb6e352ca623cf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679622966762 2023-03-24 01:56:09,795 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting edf9a9052e274ffda0fb39ec1d4dbab2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968525 2023-03-24 01:56:09,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:09,795 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,795 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f990738ba49e42978049672f623341e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968533 2023-03-24 01:56:09,796 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:09,796 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:09,797 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,797 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,798 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:09,798 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,802 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:09,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:09,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:09,803 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:09,803 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:09,803 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:09,803 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:09,806 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,806 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,806 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:09,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,807 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,809 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:09,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:09,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:09,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:09,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,811 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:09,811 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,812 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:09,812 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:09,812 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:09,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:09,829 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/28788199f1e043638e5f8f3872c8b797 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/28788199f1e043638e5f8f3872c8b797 2023-03-24 01:56:09,832 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/5cbb5af5d80943d091734ca53de38192 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5cbb5af5d80943d091734ca53de38192 2023-03-24 01:56:09,837 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 28788199f1e043638e5f8f3872c8b797(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:09,837 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:09,837 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622969790; duration=0sec 2023-03-24 01:56:09,837 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:09,837 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:09,839 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:09,839 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:09,839 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,839 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e98a3a8ae7334353a0b9e2772351aee4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.4 K 2023-03-24 01:56:09,839 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e98a3a8ae7334353a0b9e2772351aee4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1679622966113 2023-03-24 01:56:09,840 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting aa36655eba6440a187ba2162912bd5f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679622966762 2023-03-24 01:56:09,840 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 7a638180f8154839910b4596588309dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968533 2023-03-24 01:56:09,843 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 5cbb5af5d80943d091734ca53de38192(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:09,843 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:09,843 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622969791; duration=0sec 2023-03-24 01:56:09,843 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:09,846 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#467 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:09,858 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/05133a6e6f654ea1a1dbd78e585dd471 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/05133a6e6f654ea1a1dbd78e585dd471 2023-03-24 01:56:09,865 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into 05133a6e6f654ea1a1dbd78e585dd471(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:09,865 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:09,865 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622969791; duration=0sec 2023-03-24 01:56:09,865 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:09,912 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:09,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:09,916 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:09,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:09,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:09,916 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,917 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,918 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:09,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:09,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:09,918 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:09,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:09,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:09,919 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:09,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:09,920 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:09,928 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:09,928 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:09,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:09,929 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:09,929 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:09,948 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/689477a8b8304b80a6a253d747356d98 2023-03-24 01:56:09,989 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/3c023c8e4fb24fdb92c60cde604c8ecc 2023-03-24 01:56:10,034 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/d56634edefed4e15a7e8eb548f998013 2023-03-24 01:56:10,040 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/689477a8b8304b80a6a253d747356d98 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98 2023-03-24 01:56:10,044 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98, entries=150, sequenceid=529, filesize=11.9 K 2023-03-24 01:56:10,046 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/3c023c8e4fb24fdb92c60cde604c8ecc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc 2023-03-24 01:56:10,050 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc, entries=150, sequenceid=529, filesize=11.9 K 2023-03-24 01:56:10,052 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/d56634edefed4e15a7e8eb548f998013 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013 2023-03-24 01:56:10,064 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013, entries=150, sequenceid=529, filesize=11.9 K 2023-03-24 01:56:10,065 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 7db42890c5a058f4b49bb0a2fb398589 in 137ms, sequenceid=529, compaction requested=false 2023-03-24 01:56:10,065 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,065 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,070 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f564b1d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,081 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,081 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,084 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,084 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,086 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,087 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,091 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,092 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,092 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,192 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,196 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,197 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,205 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,208 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd7770c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,212 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,216 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,216 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,218 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,219 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,220 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,227 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,227 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,327 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,331 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,331 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,332 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,334 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,346 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,346 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f93ca69[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,350 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,353 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,355 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,355 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,358 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,362 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,362 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,362 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,462 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,467 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,468 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,470 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,471 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,471 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,474 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,474 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,475 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f65e582[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,485 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,488 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,488 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,588 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,592 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120657d0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,601 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,604 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,604 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,604 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,607 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,614 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,615 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,615 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,615 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,615 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,715 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,719 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,723 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,723 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:10,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,727 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,728 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@166d3d8b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,729 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,729 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,730 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,731 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,731 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,732 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,732 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,736 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,737 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,739 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,743 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,743 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,743 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:10,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:10,763 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:10,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622970784, max timestamp=1679622970799Num uniques 859; 2023-03-24 01:56:10,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:10,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622970784, max timestamp=1679622970799Num uniques 859; 2023-03-24 01:56:10,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:10,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622970784, max timestamp=1679622970799Num uniques 859; 2023-03-24 01:56:10,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:10,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623030829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623030830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623030832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,843 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,846 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,846 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,847 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,848 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:10,850 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675c6dad[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,853 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,853 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,856 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,856 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,857 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,857 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,860 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,860 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,864 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,868 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,868 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,868 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35020 deadline: 1679623030917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35048 deadline: 1679623030918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,922 DEBUG [Thread-3090] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:10,923 DEBUG [Thread-3096] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:10,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623030933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623030937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:10,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623030937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,968 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:10,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:10,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:10,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:10,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:10,971 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:10,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:10,971 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:10,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,972 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,974 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,974 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:10,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:10,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:10,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:10,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:10,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:10,975 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:10,975 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,976 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:10,976 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:10,976 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:10,976 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:10,977 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,977 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,978 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:10,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c5dd31[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:10,981 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:10,981 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,982 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:10,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:10,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:10,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:10,983 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,983 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,983 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,983 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:10,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:10,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:10,986 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:10,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:10,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:10,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:10,986 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:10,988 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,988 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:10,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:10,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:10,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:10,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:10,992 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:10,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:10,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:10,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:10,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:10,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:10,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:10,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:10,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:10,993 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:10,993 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:10,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:10,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,093 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:11,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:11,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:11,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:11,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:11,096 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:11,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:11,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:11,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,097 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,101 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:11,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:11,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:11,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:11,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:11,102 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:11,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,102 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:11,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:11,103 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:11,103 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:11,106 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,106 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17dda4ce[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:11,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:11,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,111 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:11,111 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,112 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:11,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:11,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:11,113 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:11,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:11,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:11,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:11,113 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:11,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,122 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:11,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:11,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,123 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,123 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:11,123 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:11,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:11,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623031139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623031143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623031144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/3da7611250234c8fa9f6ce5c2d4ca649 2023-03-24 01:56:11,223 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:11,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:11,226 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:11,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:11,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:11,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:11,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:11,230 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,230 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:11,230 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:11,230 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:11,230 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:11,231 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a342db[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:11,234 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:11,234 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:11,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:11,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:11,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:11,237 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,237 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:11,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:11,240 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:11,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:11,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:11,240 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:11,240 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:11,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,242 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,246 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:11,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:11,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:11,246 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,247 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,247 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:11,247 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:11,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,347 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:11,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:11,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:11,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:11,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:11,351 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:11,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:11,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:11,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,353 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:11,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:11,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:11,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:11,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:11,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,354 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:11,355 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,355 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:11,355 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:11,355 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:11,355 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:11,357 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c166b67[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:11,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:11,359 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,361 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:11,361 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:11,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,361 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:11,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:11,366 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:11,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:11,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:11,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:11,366 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:11,368 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,368 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,368 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,373 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:11,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,373 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:11,374 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:11,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:11,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623031442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623031450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623031450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,474 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:11,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:11,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:11,477 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:11,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:11,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:11,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:11,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:11,481 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,481 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:11,481 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:11,481 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:11,481 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:11,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:11,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a31f81[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:11,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:11,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,490 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,490 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,490 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:11,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:11,492 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:11,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:11,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:11,494 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:11,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:11,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:11,494 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:11,494 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:11,496 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,496 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,496 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:11,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,500 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,501 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:11,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:11,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:11,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:11,501 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:11,501 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:11,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:11,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:11,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/261b33e7adec4c71bcda4416aed94426 2023-03-24 01:56:11,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/538b72561a6a4645acd3d033d89ea65f 2023-03-24 01:56:11,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/3da7611250234c8fa9f6ce5c2d4ca649 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649 2023-03-24 01:56:11,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649, entries=150, sequenceid=544, filesize=11.9 K 2023-03-24 01:56:11,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/261b33e7adec4c71bcda4416aed94426 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426 2023-03-24 01:56:11,601 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:11,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:11,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:11,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426, entries=100, sequenceid=544, filesize=9.5 K 2023-03-24 01:56:11,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/538b72561a6a4645acd3d033d89ea65f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f 2023-03-24 01:56:11,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:11,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:11,605 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:11,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:11,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:11,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,608 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:11,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f, entries=100, sequenceid=544, filesize=9.5 K 2023-03-24 01:56:11,609 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:11,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:11,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:11,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 7db42890c5a058f4b49bb0a2fb398589 in 847ms, sequenceid=544, compaction requested=true 2023-03-24 01:56:11,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:11,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:11,610 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:11,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:11,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:11,610 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:11,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:11,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:11,611 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:11,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:11,611 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:11,611 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:11,611 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/28788199f1e043638e5f8f3872c8b797, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.5 K 2023-03-24 01:56:11,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:11,611 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:11,611 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:11,612 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:11,612 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,612 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5cbb5af5d80943d091734ca53de38192, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=34.1 K 2023-03-24 01:56:11,612 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:11,612 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:11,612 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,612 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:11,612 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 01:56:11,613 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5cbb5af5d80943d091734ca53de38192, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968533 2023-03-24 01:56:11,613 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 28788199f1e043638e5f8f3872c8b797, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968533 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:11,613 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3c023c8e4fb24fdb92c60cde604c8ecc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679622968597 2023-03-24 01:56:11,613 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:11,613 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 689477a8b8304b80a6a253d747356d98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679622968597 2023-03-24 01:56:11,614 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 261b33e7adec4c71bcda4416aed94426, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622970757 2023-03-24 01:56:11,614 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 3da7611250234c8fa9f6ce5c2d4ca649, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622970757 2023-03-24 01:56:11,625 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:11,626 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:11,638 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/a37c5974e772407e8948e3b9bad73880 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a37c5974e772407e8948e3b9bad73880 2023-03-24 01:56:11,638 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/39ed8365aad2410799ef6cb8001bd7f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39ed8365aad2410799ef6cb8001bd7f4 2023-03-24 01:56:11,649 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 39ed8365aad2410799ef6cb8001bd7f4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:11,649 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into a37c5974e772407e8948e3b9bad73880(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:11,649 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:11,649 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:11,650 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622971610; duration=0sec 2023-03-24 01:56:11,650 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622971610; duration=0sec 2023-03-24 01:56:11,650 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:11,650 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:11,650 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:11,651 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:11,651 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:11,651 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/05133a6e6f654ea1a1dbd78e585dd471, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=34.1 K 2023-03-24 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 05133a6e6f654ea1a1dbd78e585dd471, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679622968533 2023-03-24 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d56634edefed4e15a7e8eb548f998013, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679622968597 2023-03-24 01:56:11,652 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 538b72561a6a4645acd3d033d89ea65f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622970757 2023-03-24 01:56:11,657 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:11,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623031969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623031970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:11,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623031971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,023 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/c3e8f11c858e44cabb3a20fdb6d607ca 2023-03-24 01:56:12,037 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/586c9864af4d4a13b584d406e3f57fa3 2023-03-24 01:56:12,049 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/be87eefb466b4526897c2ba162dfcd38 2023-03-24 01:56:12,054 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/c3e8f11c858e44cabb3a20fdb6d607ca as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca 2023-03-24 01:56:12,057 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca, entries=150, sequenceid=579, filesize=11.9 K 2023-03-24 01:56:12,058 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/586c9864af4d4a13b584d406e3f57fa3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3 2023-03-24 01:56:12,061 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3, entries=150, sequenceid=579, filesize=11.9 K 2023-03-24 01:56:12,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/be87eefb466b4526897c2ba162dfcd38 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38 2023-03-24 01:56:12,064 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38, entries=150, sequenceid=579, filesize=11.9 K 2023-03-24 01:56:12,065 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 7db42890c5a058f4b49bb0a2fb398589 in 453ms, sequenceid=579, compaction requested=false 2023-03-24 01:56:12,065 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,065 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,065 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,067 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa22e2f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,069 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,070 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/858d52d725d444cd844aee0316c5a513 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/858d52d725d444cd844aee0316c5a513 2023-03-24 01:56:12,071 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,071 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,071 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,075 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,075 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,075 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,075 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into 858d52d725d444cd844aee0316c5a513(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:12,075 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,075 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622971610; duration=0sec 2023-03-24 01:56:12,076 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:12,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,078 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,079 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:12,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:56:12,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:12,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:12,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:12,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:12,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:12,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,085 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,086 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,085 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,086 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,086 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/1629c17970414885b85a7d7b707ceb64 2023-03-24 01:56:12,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/79f6cb3c39004539a873a2cbc7030be0 2023-03-24 01:56:12,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/bd768902768d4cc4b2d4c01428f15fe3 2023-03-24 01:56:12,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/1629c17970414885b85a7d7b707ceb64 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64 2023-03-24 01:56:12,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64, entries=200, sequenceid=593, filesize=14.3 K 2023-03-24 01:56:12,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/79f6cb3c39004539a873a2cbc7030be0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0 2023-03-24 01:56:12,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0, entries=150, sequenceid=593, filesize=11.9 K 2023-03-24 01:56:12,122 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/bd768902768d4cc4b2d4c01428f15fe3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622972112, max timestamp=1679622972121Num uniques 859; 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622972112, max timestamp=1679622972121Num uniques 859; 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679622972112, max timestamp=1679622972121Num uniques 859; 2023-03-24 01:56:12,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 01:56:12,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3, entries=150, sequenceid=593, filesize=11.9 K 2023-03-24 01:56:12,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=33.54 KB/34350 for 7db42890c5a058f4b49bb0a2fb398589 in 52ms, sequenceid=593, compaction requested=true 2023-03-24 01:56:12,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:12,133 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:12,133 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:12,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:12,133 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:12,134 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:12,134 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:12,134 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,134 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39ed8365aad2410799ef6cb8001bd7f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=38.9 K 2023-03-24 01:56:12,135 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:12,135 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 39ed8365aad2410799ef6cb8001bd7f4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622968614 2023-03-24 01:56:12,135 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:12,135 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,136 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a37c5974e772407e8948e3b9bad73880, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.6 K 2023-03-24 01:56:12,136 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c3e8f11c858e44cabb3a20fdb6d607ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679622970824 2023-03-24 01:56:12,136 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a37c5974e772407e8948e3b9bad73880, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622968614 2023-03-24 01:56:12,136 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1629c17970414885b85a7d7b707ceb64, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679622971962 2023-03-24 01:56:12,136 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 586c9864af4d4a13b584d406e3f57fa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679622970824 2023-03-24 01:56:12,137 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 79f6cb3c39004539a873a2cbc7030be0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679622971962 2023-03-24 01:56:12,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:12,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=53.67 KB heapSize=141.23 KB 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:12,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,161 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:12,166 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:12,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/c73c7eca836f4cf7be2ba9edc297efcc 2023-03-24 01:56:12,186 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/bd80f0239a294764805d9cf78309ab29 2023-03-24 01:56:12,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,190 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,192 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,192 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,194 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,194 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,194 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,194 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,195 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7de5009a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,197 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,197 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,205 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,205 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,208 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/fddc90d5449648f3b6205cd85ec31e1f 2023-03-24 01:56:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,210 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,213 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,213 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/c73c7eca836f4cf7be2ba9edc297efcc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc 2023-03-24 01:56:12,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc, entries=250, sequenceid=617, filesize=16.7 K 2023-03-24 01:56:12,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/bd80f0239a294764805d9cf78309ab29 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29 2023-03-24 01:56:12,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622972186, max timestamp=1679622972219Num uniques 859; 2023-03-24 01:56:12,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:12,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29, entries=150, sequenceid=617, filesize=11.9 K 2023-03-24 01:56:12,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622972186, max timestamp=1679622972219Num uniques 859; 2023-03-24 01:56:12,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:12,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/fddc90d5449648f3b6205cd85ec31e1f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f 2023-03-24 01:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622972186, max timestamp=1679622972219Num uniques 859; 2023-03-24 01:56:12,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:12,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f, entries=150, sequenceid=617, filesize=11.9 K 2023-03-24 01:56:12,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~141.19 KB/144576, currentSize=40.25 KB/41220 for 7db42890c5a058f4b49bb0a2fb398589 in 72ms, sequenceid=617, compaction requested=true 2023-03-24 01:56:12,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:12,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 01:56:12,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-24 01:56:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:12,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=60.38 KB heapSize=156.47 KB 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:12,233 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/b87723582f6a433aa6c54d09f1c01be0 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622972268, max timestamp=1679622972298Num uniques 859; 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622972268, max timestamp=1679622972298Num uniques 859; 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679622972268, max timestamp=1679622972298Num uniques 859; 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:12,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 01:56:12,313 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,321 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,321 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,322 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,322 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,322 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,332 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22770112[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,334 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,334 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,340 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,342 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,342 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,343 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,346 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,348 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,348 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,349 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,349 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,352 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623032352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623032361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623032362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,452 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,456 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,465 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc54dca[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,469 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,471 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,473 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,474 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623032465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623032468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,479 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,483 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,484 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,484 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,484 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623032479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,581 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/97143348f6c44f63a118f967bde0e5ca as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/97143348f6c44f63a118f967bde0e5ca 2023-03-24 01:56:12,582 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/0b6e20def3264322acc7507867c11878 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/0b6e20def3264322acc7507867c11878 2023-03-24 01:56:12,584 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,586 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 97143348f6c44f63a118f967bde0e5ca(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:12,586 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,586 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622972133; duration=0sec 2023-03-24 01:56:12,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-24 01:56:12,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-03-24 01:56:12,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:12,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:12,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:12,588 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,589 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,589 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 0b6e20def3264322acc7507867c11878(size=12.9 K), total size for store is 29.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:12,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,589 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,589 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,590 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622972133; duration=0sec 2023-03-24 01:56:12,590 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:12,590 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:12,590 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:12,590 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,590 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/858d52d725d444cd844aee0316c5a513, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=48.5 K 2023-03-24 01:56:12,590 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:12,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 858d52d725d444cd844aee0316c5a513, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1679622968614 2023-03-24 01:56:12,591 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:56:12,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,591 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,591 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting be87eefb466b4526897c2ba162dfcd38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1679622970824 2023-03-24 01:56:12,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,592 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting bd768902768d4cc4b2d4c01428f15fe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679622971962 2023-03-24 01:56:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,592 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:12,592 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:12,593 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:12,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,592 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting fddc90d5449648f3b6205cd85ec31e1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679622972114 2023-03-24 01:56:12,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,593 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,594 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,594 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5108d14b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,600 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,604 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,604 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:12,605 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,605 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,615 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,617 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/dbaca42bc1724edabda106e24e191c2f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dbaca42bc1724edabda106e24e191c2f 2023-03-24 01:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,619 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,619 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,622 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,622 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,624 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into dbaca42bc1724edabda106e24e191c2f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:12,624 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,624 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=12, startTime=1679622972227; duration=0sec 2023-03-24 01:56:12,624 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:12,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/703ed3809327457c80fa389d202beaeb 2023-03-24 01:56:12,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/69e6c627bbff44faa20e5696ad128d0a 2023-03-24 01:56:12,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/b87723582f6a433aa6c54d09f1c01be0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0 2023-03-24 01:56:12,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0, entries=250, sequenceid=641, filesize=16.7 K 2023-03-24 01:56:12,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/703ed3809327457c80fa389d202beaeb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb 2023-03-24 01:56:12,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb, entries=150, sequenceid=641, filesize=11.9 K 2023-03-24 01:56:12,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/69e6c627bbff44faa20e5696ad128d0a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a 2023-03-24 01:56:12,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623032680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623032682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a, entries=150, sequenceid=641, filesize=11.9 K 2023-03-24 01:56:12,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~156.42 KB/160176, currentSize=140.89 KB/144270 for 7db42890c5a058f4b49bb0a2fb398589 in 460ms, sequenceid=641, compaction requested=true 2023-03-24 01:56:12,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:12,693 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:12,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:12,693 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:12,693 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:12,694 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:12,694 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:12,694 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,694 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/0b6e20def3264322acc7507867c11878, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=46.2 K 2023-03-24 01:56:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:12,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:12,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0b6e20def3264322acc7507867c11878, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679622971962 2023-03-24 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:12,696 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,696 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/97143348f6c44f63a118f967bde0e5ca, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.7 K 2023-03-24 01:56:12,696 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting c73c7eca836f4cf7be2ba9edc297efcc, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679622972114 2023-03-24 01:56:12,697 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 97143348f6c44f63a118f967bde0e5ca, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1679622971962 2023-03-24 01:56:12,698 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b87723582f6a433aa6c54d09f1c01be0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679622972212 2023-03-24 01:56:12,698 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting bd80f0239a294764805d9cf78309ab29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679622972114 2023-03-24 01:56:12,700 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 703ed3809327457c80fa389d202beaeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679622972228 2023-03-24 01:56:12,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/3710196d69a549ae9d31aea9c6fbebf6 2023-03-24 01:56:12,707 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:12,710 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:12,723 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,728 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,732 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/3c4b5678a23041aeb1993e2ed434dd10 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c4b5678a23041aeb1993e2ed434dd10 2023-03-24 01:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,734 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,734 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,734 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,734 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,734 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,736 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 3c4b5678a23041aeb1993e2ed434dd10(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:12,736 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:12,736 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=13, startTime=1679622972693; duration=0sec 2023-03-24 01:56:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,736 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:12,736 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43cfa20[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,737 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,737 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:12,738 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:12,738 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:12,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,739 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,740 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,743 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,743 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623032741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,745 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,747 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,754 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,754 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,754 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,754 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,754 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623032847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,854 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,857 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,858 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,858 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,859 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,861 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,861 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,861 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,861 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,863 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed91424[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,865 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,865 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,871 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,871 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:12,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,878 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:12,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:12,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,879 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:12,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,879 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,879 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:12,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:12,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:12,979 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:12,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:12,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:12,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:12,982 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:12,982 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:12,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:12,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,983 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:12,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:12,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:12,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:12,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:12,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:12,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:12,989 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:12,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:12,991 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:12,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b97b128[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:12,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:12,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,994 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:12,995 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:12,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:12,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623032991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:12,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623032992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:12,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:12,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:12,999 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:12,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:12,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:12,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:12,999 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:13,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,007 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:13,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,007 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:13,008 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:13,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:13,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623033052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,108 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:13,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:13,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:13,111 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:13,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:13,111 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:13,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,112 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,113 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:13,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:13,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:13,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:13,114 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:13,115 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,115 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:13,115 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:13,115 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:13,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:13,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:13,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c82b598[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:13,119 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:13,119 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,122 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,122 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:13,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:13,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:13,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:13,124 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/056a164f8b03418988a299d23dd1d4ee as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/056a164f8b03418988a299d23dd1d4ee 2023-03-24 01:56:13,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:13,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,128 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 056a164f8b03418988a299d23dd1d4ee(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:13,128 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:13,128 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=13, startTime=1679622972693; duration=0sec 2023-03-24 01:56:13,128 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:13,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:13,129 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:13,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:13,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:13,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:13,129 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:13,130 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,130 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,131 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,132 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,136 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e365c26db3714364bbf514ded991d8d0 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:13,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,138 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:13,138 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:13,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,137 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:13,238 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:13,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:13,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:13,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:13,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:13,241 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:13,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:13,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:13,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:13,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:13,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:13,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:13,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:13,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:13,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:13,248 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:13,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8cbfd2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:13,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:13,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,252 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:13,253 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:13,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:13,254 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:13,254 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:13,254 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:13,255 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,256 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,256 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:13,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:13,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,260 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,261 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:13,261 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:13,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,361 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:13,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:13,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:13,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:13,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:13,364 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,364 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:13,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35074 deadline: 1679623033358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,366 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,366 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:13,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:13,368 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,368 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:13,368 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:13,368 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:13,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,370 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59a2f233[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:13,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:13,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,372 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:13,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:13,373 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,373 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:13,374 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,374 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,374 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,374 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:13,376 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:13,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:13,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:13,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:13,376 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:13,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,383 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:13,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:13,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,384 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:13,384 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:13,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:13,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,484 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:13,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:13,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:13,487 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:13,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,487 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:13,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:13,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,489 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,490 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,490 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:13,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:13,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:13,491 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,491 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:13,491 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:13,491 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:13,491 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:13,493 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,493 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,493 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,493 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7a16de[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:13,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:13,496 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,497 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:13,497 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:13,497 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:13,499 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:13,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35036 deadline: 1679623033497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,502 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:13,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:13,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:13,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:13,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:13,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35054 deadline: 1679623033499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:13,502 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:13,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:13,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,504 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:13,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:13,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,506 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:13,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:13,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:13,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:13,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:13,508 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:13,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:13,509 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:13,509 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:13,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=680 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/d85c0e06a4f44bbab54f15c70134eae8 2023-03-24 01:56:13,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/3710196d69a549ae9d31aea9c6fbebf6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6 2023-03-24 01:56:13,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6, entries=200, sequenceid=680, filesize=14.3 K 2023-03-24 01:56:13,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/e365c26db3714364bbf514ded991d8d0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0 2023-03-24 01:56:13,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0, entries=150, sequenceid=680, filesize=11.9 K 2023-03-24 01:56:13,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/d85c0e06a4f44bbab54f15c70134eae8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8 2023-03-24 01:56:13,564 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8, entries=150, sequenceid=680, filesize=11.9 K 2023-03-24 01:56:13,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 7db42890c5a058f4b49bb0a2fb398589 in 870ms, sequenceid=680, compaction requested=true 2023-03-24 01:56:13,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:13,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:13,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:13,565 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:56:13,565 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:13,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:13,566 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:13,566 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:13,566 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:13,566 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:13,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:13,567 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dbaca42bc1724edabda106e24e191c2f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.7 K 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. because compaction request was cancelled 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting dbaca42bc1724edabda106e24e191c2f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=617, earliestPutTs=1679622972114 2023-03-24 01:56:13,567 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 69e6c627bbff44faa20e5696ad128d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679622972228 2023-03-24 01:56:13,568 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting d85c0e06a4f44bbab54f15c70134eae8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1679622972359 2023-03-24 01:56:13,575 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:13,584 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/1d055d43efb64785a65679a21eec943d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1d055d43efb64785a65679a21eec943d 2023-03-24 01:56:13,587 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into 1d055d43efb64785a65679a21eec943d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:13,587 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:13,587 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622973565; duration=0sec 2023-03-24 01:56:13,588 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:13,609 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:13,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:13,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:13,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:13,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:13,612 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:13,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:13,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:13,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:13,622 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:13,622 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:13,623 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:13,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:13,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:13,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:13,625 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:13,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:13,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:13,625 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 01:56:13,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:13,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:13,625 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:13,626 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:13,626 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:13,626 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:13,634 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/ed86daf10aad4341b24f144562df732c 2023-03-24 01:56:13,652 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/f85aca3426fc433d991d77fa4091a626 2023-03-24 01:56:13,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:13,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:13,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622973909, max timestamp=1679622973960Num uniques 859; 2023-03-24 01:56:13,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622973909, max timestamp=1679622973960Num uniques 859; 2023-03-24 01:56:13,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:13,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622973909, max timestamp=1679622973960Num uniques 859; 2023-03-24 01:56:13,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:13,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622974015, max timestamp=1679622974052Num uniques 1018; 2023-03-24 01:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622974015, max timestamp=1679622974052Num uniques 1018; 2023-03-24 01:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679622974015, max timestamp=1679622974052Num uniques 1018; 2023-03-24 01:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:56:14,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 01:56:14,066 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/e4fd74d2436746fe93eaffdf572f7d76 2023-03-24 01:56:14,070 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/ed86daf10aad4341b24f144562df732c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c 2023-03-24 01:56:14,073 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c, entries=150, sequenceid=692, filesize=11.9 K 2023-03-24 01:56:14,074 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/f85aca3426fc433d991d77fa4091a626 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626 2023-03-24 01:56:14,077 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626, entries=150, sequenceid=692, filesize=11.9 K 2023-03-24 01:56:14,077 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/e4fd74d2436746fe93eaffdf572f7d76 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76 2023-03-24 01:56:14,081 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76, entries=150, sequenceid=692, filesize=11.9 K 2023-03-24 01:56:14,081 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=46.96 KB/48090 for 7db42890c5a058f4b49bb0a2fb398589 in 456ms, sequenceid=692, compaction requested=true 2023-03-24 01:56:14,082 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:14,082 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:14,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-03-24 01:56:14,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:14,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:14,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:14,086 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ad5edc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:14,087 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:14,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,088 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,088 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:14,092 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:14,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:14,096 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:14,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:14,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:14,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:14,096 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:14,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/a24ee0724f0f44a198c2c78153f3bd49 2023-03-24 01:56:14,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,098 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,100 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,108 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:14,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:14,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:14,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:14,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:14,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:14,109 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:14,109 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:14,209 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:14,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:14,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:14,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:14,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:14,212 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:14,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:14,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:14,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:14,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,213 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,214 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:14,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:14,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:14,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:14,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:14,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:14,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. started... 2023-03-24 01:56:14,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. as already flushing 2023-03-24 01:56:14,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622974164, max timestamp=1679622974214Num uniques 859; 2023-03-24 01:56:14,216 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622974164, max timestamp=1679622974214Num uniques 859; 2023-03-24 01:56:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:14,217 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:14,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:14,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622974164, max timestamp=1679622974214Num uniques 859; 2023-03-24 01:56:14,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:14,219 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,219 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,220 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:14,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb50ff4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:14,221 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:14,221 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,223 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:14,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:14,223 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:14,224 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,224 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:14,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:14,227 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:14,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:14,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:14,227 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:14,227 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:14,228 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,229 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:14,229 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,229 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:14,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:14,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:14,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:14,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:14,234 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:14,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:14,235 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:14,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:14,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:14,235 DEBUG [Thread-3098] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:14,235 DEBUG [Thread-3098] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:14,271 DEBUG [Thread-3103] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:62686 2023-03-24 01:56:14,271 DEBUG [Thread-3103] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,273 DEBUG [Thread-3092] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:62686 2023-03-24 01:56:14,273 DEBUG [Thread-3092] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,274 DEBUG [Thread-3107] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:62686 2023-03-24 01:56:14,274 DEBUG [Thread-3107] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,280 DEBUG [Thread-3099] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:62686 2023-03-24 01:56:14,280 DEBUG [Thread-3099] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,285 DEBUG [Thread-3101] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x461f5f3f to 127.0.0.1:62686 2023-03-24 01:56:14,285 DEBUG [Thread-3101] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,287 DEBUG [Thread-3105] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:62686 2023-03-24 01:56:14,287 DEBUG [Thread-3105] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,335 DEBUG [Thread-3098] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:14,505 DEBUG [Thread-3088] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df63a4 to 127.0.0.1:62686 2023-03-24 01:56:14,505 DEBUG [Thread-3088] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,511 DEBUG [Thread-3094] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:62686 2023-03-24 01:56:14,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/119580167e4f46e5a0682a269dd67d6b 2023-03-24 01:56:14,511 DEBUG [Thread-3094] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/1cd8b82fe08344d3995b367e1dbab5bd 2023-03-24 01:56:14,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/a24ee0724f0f44a198c2c78153f3bd49 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49 2023-03-24 01:56:14,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49, entries=200, sequenceid=734, filesize=14.3 K 2023-03-24 01:56:14,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/119580167e4f46e5a0682a269dd67d6b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b 2023-03-24 01:56:14,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b, entries=150, sequenceid=734, filesize=11.9 K 2023-03-24 01:56:14,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/1cd8b82fe08344d3995b367e1dbab5bd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd 2023-03-24 01:56:14,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd, entries=150, sequenceid=734, filesize=11.9 K 2023-03-24 01:56:14,933 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=93.93 KB/96180 for 7db42890c5a058f4b49bb0a2fb398589 in 847ms, sequenceid=734, compaction requested=true 2023-03-24 01:56:14,933 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:14,933 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:14,933 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:14,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:14,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:14,934 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:14,935 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54714 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:14,935 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/A is initiating minor compaction (all files) 2023-03-24 01:56:14,935 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:14,935 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/A in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,935 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/B is initiating minor compaction (all files) 2023-03-24 01:56:14,935 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/056a164f8b03418988a299d23dd1d4ee, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=53.4 K 2023-03-24 01:56:14,935 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/B in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,935 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c4b5678a23041aeb1993e2ed434dd10, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=48.7 K 2023-03-24 01:56:14,936 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3c4b5678a23041aeb1993e2ed434dd10, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679622972228 2023-03-24 01:56:14,936 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 056a164f8b03418988a299d23dd1d4ee, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1679622972228 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e365c26db3714364bbf514ded991d8d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1679622972359 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 3710196d69a549ae9d31aea9c6fbebf6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1679622972311 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f85aca3426fc433d991d77fa4091a626, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679622972715 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ed86daf10aad4341b24f144562df732c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679622972715 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 119580167e4f46e5a0682a269dd67d6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1679622974062 2023-03-24 01:56:14,937 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting a24ee0724f0f44a198c2c78153f3bd49, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1679622974052 2023-03-24 01:56:14,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:14,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=100.63 KB heapSize=285.38 KB 2023-03-24 01:56:14,944 DEBUG [Thread-3096] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:62686 2023-03-24 01:56:14,944 DEBUG [Thread-3096] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:14,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:14,944 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#B#compaction#505 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:14,944 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#A#compaction#504 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:14,946 DEBUG [Thread-3090] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:62686 2023-03-24 01:56:14,946 DEBUG [Thread-3090] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,946 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 210 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1745 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5234 rows 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1752 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5256 rows 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1747 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5241 rows 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1749 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5243 rows 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(402): scanned 1752 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(403): verified 5256 rows 2023-03-24 01:56:14,947 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:56:14,947 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681750df to 127.0.0.1:62686 2023-03-24 01:56:14,948 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:14,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/bb5bcc6f270349ffbaa81ce5aab81257 2023-03-24 01:56:14,960 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:56:14,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:56:14,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:14,962 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/49470ac04e2c4829a68dfbfa28f75929 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/49470ac04e2c4829a68dfbfa28f75929 2023-03-24 01:56:14,964 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/5f52e584cde74140b4de7a0e19d2f124 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/5f52e584cde74140b4de7a0e19d2f124 2023-03-24 01:56:14,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:14,965 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622974965"}]},"ts":"1679622974965"} 2023-03-24 01:56:14,966 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:56:14,967 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:56:14,968 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, UNASSIGN}] 2023-03-24 01:56:14,968 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/B of 7db42890c5a058f4b49bb0a2fb398589 into 49470ac04e2c4829a68dfbfa28f75929(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:14,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:14,968 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/B, priority=12, startTime=1679622974934; duration=0sec 2023-03-24 01:56:14,968 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/A of 7db42890c5a058f4b49bb0a2fb398589 into 5f52e584cde74140b4de7a0e19d2f124(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:14,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:14,968 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:14,968 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/A, priority=12, startTime=1679622974933; duration=0sec 2023-03-24 01:56:14,968 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:14,968 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, UNASSIGN 2023-03-24 01:56:14,968 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:14,969 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=7db42890c5a058f4b49bb0a2fb398589, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:14,969 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622974969"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622974969"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622974969"}]},"ts":"1679622974969"} 2023-03-24 01:56:14,969 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:14,969 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 7db42890c5a058f4b49bb0a2fb398589/C is initiating minor compaction (all files) 2023-03-24 01:56:14,970 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7db42890c5a058f4b49bb0a2fb398589/C in TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:14,970 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1d055d43efb64785a65679a21eec943d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp, totalSize=36.8 K 2023-03-24 01:56:14,970 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1d055d43efb64785a65679a21eec943d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=680, earliestPutTs=1679622972359 2023-03-24 01:56:14,970 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e4fd74d2436746fe93eaffdf572f7d76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1679622972715 2023-03-24 01:56:14,971 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1cd8b82fe08344d3995b367e1dbab5bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1679622974062 2023-03-24 01:56:14,972 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:56:14,977 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 7db42890c5a058f4b49bb0a2fb398589#C#compaction#508 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:15,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:15,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:15,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7db42890c5a058f4b49bb0a2fb398589, disabling compactions & flushes 2023-03-24 01:56:15,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:15,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/8fc26a7828844fcea2589b35eec89573 2023-03-24 01:56:15,384 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/a5866780e07c4fb6a9d993cf0a0689ea as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a5866780e07c4fb6a9d993cf0a0689ea 2023-03-24 01:56:15,387 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7db42890c5a058f4b49bb0a2fb398589/C of 7db42890c5a058f4b49bb0a2fb398589 into a5866780e07c4fb6a9d993cf0a0689ea(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:15,387 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:15,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:15,387 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589., storeName=7db42890c5a058f4b49bb0a2fb398589/C, priority=13, startTime=1679622974934; duration=0sec 2023-03-24 01:56:15,387 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:15,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/03ba016bdbee44bab0c27480ba693b06 2023-03-24 01:56:15,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/bb5bcc6f270349ffbaa81ce5aab81257 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/bb5bcc6f270349ffbaa81ce5aab81257 2023-03-24 01:56:15,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/bb5bcc6f270349ffbaa81ce5aab81257, entries=150, sequenceid=766, filesize=11.9 K 2023-03-24 01:56:15,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/8fc26a7828844fcea2589b35eec89573 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/8fc26a7828844fcea2589b35eec89573 2023-03-24 01:56:15,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/8fc26a7828844fcea2589b35eec89573, entries=150, sequenceid=766, filesize=11.9 K 2023-03-24 01:56:15,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/03ba016bdbee44bab0c27480ba693b06 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/03ba016bdbee44bab0c27480ba693b06 2023-03-24 01:56:15,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/03ba016bdbee44bab0c27480ba693b06, entries=150, sequenceid=766, filesize=11.9 K 2023-03-24 01:56:15,796 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~285.33 KB/292176, currentSize=6.71 KB/6870 for 7db42890c5a058f4b49bb0a2fb398589 in 853ms, sequenceid=766, compaction requested=false 2023-03-24 01:56:15,796 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:15,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. after waiting 0 ms 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:15,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7db42890c5a058f4b49bb0a2fb398589 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=A 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=B 2023-03-24 01:56:15,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:15,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7db42890c5a058f4b49bb0a2fb398589, store=C 2023-03-24 01:56:15,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:16,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:16,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/a0c343f169b649d59ff4d6a796096ace 2023-03-24 01:56:16,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/658bf0329b7049cc9244f1e4319ed15a 2023-03-24 01:56:17,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/087cf1ca4e2d4533b3c72de9ed55dd17 2023-03-24 01:56:17,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/A/a0c343f169b649d59ff4d6a796096ace as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a0c343f169b649d59ff4d6a796096ace 2023-03-24 01:56:17,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a0c343f169b649d59ff4d6a796096ace, entries=50, sequenceid=773, filesize=7.1 K 2023-03-24 01:56:17,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/B/658bf0329b7049cc9244f1e4319ed15a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/658bf0329b7049cc9244f1e4319ed15a 2023-03-24 01:56:17,034 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/658bf0329b7049cc9244f1e4319ed15a, entries=50, sequenceid=773, filesize=7.1 K 2023-03-24 01:56:17,034 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/.tmp/C/087cf1ca4e2d4533b3c72de9ed55dd17 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/087cf1ca4e2d4533b3c72de9ed55dd17 2023-03-24 01:56:17,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/087cf1ca4e2d4533b3c72de9ed55dd17, entries=50, sequenceid=773, filesize=7.1 K 2023-03-24 01:56:17,038 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for 7db42890c5a058f4b49bb0a2fb398589 in 1240ms, sequenceid=773, compaction requested=true 2023-03-24 01:56:17,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4ad85f30f9194fe796c487d2b83dfe79, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/21e3af9d26854ab88bc2b9acf5f03b72, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c5d7ead7b88246dd8f06f4be92e4b028, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39c52d2c036a4ad79eb95523ba50faba, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/03517cda1e3c4ad79d53359913493e7b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/be381b6bd2a34211b313fa013884150b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/28788199f1e043638e5f8f3872c8b797, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39ed8365aad2410799ef6cb8001bd7f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/0b6e20def3264322acc7507867c11878, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/056a164f8b03418988a299d23dd1d4ee, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49] to archive 2023-03-24 01:56:17,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:17,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/58a4422791a54a7c89c5fc95610478f1 2023-03-24 01:56:17,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f67166ba6d31470bbade6399a134080e 2023-03-24 01:56:17,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4ad85f30f9194fe796c487d2b83dfe79 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4ad85f30f9194fe796c487d2b83dfe79 2023-03-24 01:56:17,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/792c7e3e2af24f42ad81c5553ac7c452 2023-03-24 01:56:17,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f1eb79572a48463eb3e0000b34b1d074 2023-03-24 01:56:17,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/feaf7ed6efdf474ea240b11f8e793745 2023-03-24 01:56:17,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/21e3af9d26854ab88bc2b9acf5f03b72 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/21e3af9d26854ab88bc2b9acf5f03b72 2023-03-24 01:56:17,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ec78bd9b10a04f8da051d11eab51c624 2023-03-24 01:56:17,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/34828f41196747e996f5ed4e0b45f884 2023-03-24 01:56:17,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1f8b7f185d194879b0a5590b79f77d2f 2023-03-24 01:56:17,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c5d7ead7b88246dd8f06f4be92e4b028 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c5d7ead7b88246dd8f06f4be92e4b028 2023-03-24 01:56:17,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/40065a74d27a41c48f012be66b0a3041 2023-03-24 01:56:17,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/33ca2e3e89e74d67a3dbc74426519909 2023-03-24 01:56:17,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39c52d2c036a4ad79eb95523ba50faba to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39c52d2c036a4ad79eb95523ba50faba 2023-03-24 01:56:17,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/e9cc5db549624d4e8620ae265e9a07bb 2023-03-24 01:56:17,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/529336c141c646518941ed7b13176602 2023-03-24 01:56:17,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b1ceb8bffeb442d7b0e6e46891af23aa 2023-03-24 01:56:17,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/03517cda1e3c4ad79d53359913493e7b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/03517cda1e3c4ad79d53359913493e7b 2023-03-24 01:56:17,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/4711b84b025a4fcd9f86ce08f17fd12a 2023-03-24 01:56:17,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/f8e5e4a1dd764bb2abdda7377bb62a9a 2023-03-24 01:56:17,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/be381b6bd2a34211b313fa013884150b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/be381b6bd2a34211b313fa013884150b 2023-03-24 01:56:17,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/8828cb2b4ff145b4b5de30331eff220b 2023-03-24 01:56:17,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/edf9a9052e274ffda0fb39ec1d4dbab2 2023-03-24 01:56:17,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/28788199f1e043638e5f8f3872c8b797 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/28788199f1e043638e5f8f3872c8b797 2023-03-24 01:56:17,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/689477a8b8304b80a6a253d747356d98 2023-03-24 01:56:17,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39ed8365aad2410799ef6cb8001bd7f4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/39ed8365aad2410799ef6cb8001bd7f4 2023-03-24 01:56:17,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3da7611250234c8fa9f6ce5c2d4ca649 2023-03-24 01:56:17,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c3e8f11c858e44cabb3a20fdb6d607ca 2023-03-24 01:56:17,067 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/1629c17970414885b85a7d7b707ceb64 2023-03-24 01:56:17,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/0b6e20def3264322acc7507867c11878 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/0b6e20def3264322acc7507867c11878 2023-03-24 01:56:17,069 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/c73c7eca836f4cf7be2ba9edc297efcc 2023-03-24 01:56:17,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:17,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/b87723582f6a433aa6c54d09f1c01be0 2023-03-24 01:56:17,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/056a164f8b03418988a299d23dd1d4ee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/056a164f8b03418988a299d23dd1d4ee 2023-03-24 01:56:17,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/3710196d69a549ae9d31aea9c6fbebf6 2023-03-24 01:56:17,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/ed86daf10aad4341b24f144562df732c 2023-03-24 01:56:17,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a24ee0724f0f44a198c2c78153f3bd49 2023-03-24 01:56:17,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/221824cef6f543b7a994b652b4006fa2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/c18be2c5e3b54e0a957ff531235b2c6c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/526114cb8eff4feb981c8b3702e898cb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/53816ff13f0546249a38b39b12863d11, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e5a846e7fd284a289fd1fbbbc8048fb8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/228caa53734a4eb2a2410ff246923497, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5cbb5af5d80943d091734ca53de38192, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a37c5974e772407e8948e3b9bad73880, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/97143348f6c44f63a118f967bde0e5ca, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c4b5678a23041aeb1993e2ed434dd10, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b] to archive 2023-03-24 01:56:17,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:17,079 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/467b7b3fb87a475b8d0666afe35895db 2023-03-24 01:56:17,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/b416c03c6aa64b1ebab5ebd519cee7d6 2023-03-24 01:56:17,081 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/221824cef6f543b7a994b652b4006fa2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/221824cef6f543b7a994b652b4006fa2 2023-03-24 01:56:17,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/88bae949e5784f368dda9a465fdfdb8d 2023-03-24 01:56:17,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/4b6ae166392c48c99df5cac4618dc8b0 2023-03-24 01:56:17,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/c18be2c5e3b54e0a957ff531235b2c6c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/c18be2c5e3b54e0a957ff531235b2c6c 2023-03-24 01:56:17,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e97bb21fb74c4e3b93e582552e9331ea 2023-03-24 01:56:17,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/61ecc4d003924826862322785fa6cefa 2023-03-24 01:56:17,086 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5d94dcee70d147819e26df86cfa19aee 2023-03-24 01:56:17,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/526114cb8eff4feb981c8b3702e898cb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/526114cb8eff4feb981c8b3702e898cb 2023-03-24 01:56:17,088 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/7aa0ba12f9094876a0a0fd514b57e259 2023-03-24 01:56:17,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/0a2092e8fbfb46fe98cfbc1b975b7320 2023-03-24 01:56:17,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/53816ff13f0546249a38b39b12863d11 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/53816ff13f0546249a38b39b12863d11 2023-03-24 01:56:17,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/1908fa44450c4fe890d6235b2a5ba39e 2023-03-24 01:56:17,091 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/29a075c10e5943da959b1a1135b4aac9 2023-03-24 01:56:17,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/9a250d89cc8e4b0c98b198b4d411174e 2023-03-24 01:56:17,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e5a846e7fd284a289fd1fbbbc8048fb8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e5a846e7fd284a289fd1fbbbc8048fb8 2023-03-24 01:56:17,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a5528378ec3646b4956290cfd50beefe 2023-03-24 01:56:17,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/479b0019814b4d51815e1588555d8099 2023-03-24 01:56:17,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/228caa53734a4eb2a2410ff246923497 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/228caa53734a4eb2a2410ff246923497 2023-03-24 01:56:17,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e25d93cd813b45a3ba718bdb32a8c87b 2023-03-24 01:56:17,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e33120abf59b43ccbbb6e352ca623cf9 2023-03-24 01:56:17,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5cbb5af5d80943d091734ca53de38192 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/5cbb5af5d80943d091734ca53de38192 2023-03-24 01:56:17,098 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f990738ba49e42978049672f623341e5 2023-03-24 01:56:17,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c023c8e4fb24fdb92c60cde604c8ecc 2023-03-24 01:56:17,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a37c5974e772407e8948e3b9bad73880 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/a37c5974e772407e8948e3b9bad73880 2023-03-24 01:56:17,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/261b33e7adec4c71bcda4416aed94426 2023-03-24 01:56:17,101 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/586c9864af4d4a13b584d406e3f57fa3 2023-03-24 01:56:17,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/97143348f6c44f63a118f967bde0e5ca to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/97143348f6c44f63a118f967bde0e5ca 2023-03-24 01:56:17,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/79f6cb3c39004539a873a2cbc7030be0 2023-03-24 01:56:17,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/bd80f0239a294764805d9cf78309ab29 2023-03-24 01:56:17,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c4b5678a23041aeb1993e2ed434dd10 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/3c4b5678a23041aeb1993e2ed434dd10 2023-03-24 01:56:17,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/703ed3809327457c80fa389d202beaeb 2023-03-24 01:56:17,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/e365c26db3714364bbf514ded991d8d0 2023-03-24 01:56:17,107 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/f85aca3426fc433d991d77fa4091a626 2023-03-24 01:56:17,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/119580167e4f46e5a0682a269dd67d6b 2023-03-24 01:56:17,111 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a4822353da624dcebdcd99b06d2e67ab, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/c303b96cd2bf45bab4bd3f3f44744e26, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9574fd792f5742aa916b021db5a5df2a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/561e60e02fbd45878070a4b5b4d15605, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e250938696ef42f6b6c2d9faf72e421d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e98a3a8ae7334353a0b9e2772351aee4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/05133a6e6f654ea1a1dbd78e585dd471, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/858d52d725d444cd844aee0316c5a513, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dbaca42bc1724edabda106e24e191c2f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1d055d43efb64785a65679a21eec943d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd] to archive 2023-03-24 01:56:17,112 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:17,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/76dcc33093554c0fa17ef347c09287ab 2023-03-24 01:56:17,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/b620bc81f8244377b9894aaf8afe6ebe 2023-03-24 01:56:17,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a4822353da624dcebdcd99b06d2e67ab to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a4822353da624dcebdcd99b06d2e67ab 2023-03-24 01:56:17,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6c545aacdf6b437484162f08a6b574b3 2023-03-24 01:56:17,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/143c204805704d93a5ea1a6c9cc2e25e 2023-03-24 01:56:17,117 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/c303b96cd2bf45bab4bd3f3f44744e26 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/c303b96cd2bf45bab4bd3f3f44744e26 2023-03-24 01:56:17,117 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/813c33a63c3c4e66b1de8deb64e3071e 2023-03-24 01:56:17,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9bef35ad07ef460781cb8d81e42e7d03 2023-03-24 01:56:17,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/6ae80da2b7e04f7abf62063c36f6e10b 2023-03-24 01:56:17,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9574fd792f5742aa916b021db5a5df2a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/9574fd792f5742aa916b021db5a5df2a 2023-03-24 01:56:17,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dae9dcc974dc4d71a59272328f751e04 2023-03-24 01:56:17,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ed7c7978b759412f92bbd317cf209d8b 2023-03-24 01:56:17,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/561e60e02fbd45878070a4b5b4d15605 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/561e60e02fbd45878070a4b5b4d15605 2023-03-24 01:56:17,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/eae6877f863d4170a2c0833442a128d2 2023-03-24 01:56:17,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/ecef18fc03fb4134851cee3f13bb6e78 2023-03-24 01:56:17,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/2c1707b74a0348ec8ef9e6fb3c991896 2023-03-24 01:56:17,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e250938696ef42f6b6c2d9faf72e421d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e250938696ef42f6b6c2d9faf72e421d 2023-03-24 01:56:17,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bdec12ab933c40d1bc6e4b3328e6d172 2023-03-24 01:56:17,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/53b33ad535ce485991fd411785a17f0c 2023-03-24 01:56:17,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e98a3a8ae7334353a0b9e2772351aee4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e98a3a8ae7334353a0b9e2772351aee4 2023-03-24 01:56:17,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/73d5dd1a616745fbb2aab01785b5c793 2023-03-24 01:56:17,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/aa36655eba6440a187ba2162912bd5f7 2023-03-24 01:56:17,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/05133a6e6f654ea1a1dbd78e585dd471 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/05133a6e6f654ea1a1dbd78e585dd471 2023-03-24 01:56:17,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/7a638180f8154839910b4596588309dc 2023-03-24 01:56:17,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d56634edefed4e15a7e8eb548f998013 2023-03-24 01:56:17,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/858d52d725d444cd844aee0316c5a513 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/858d52d725d444cd844aee0316c5a513 2023-03-24 01:56:17,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/538b72561a6a4645acd3d033d89ea65f 2023-03-24 01:56:17,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/be87eefb466b4526897c2ba162dfcd38 2023-03-24 01:56:17,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/bd768902768d4cc4b2d4c01428f15fe3 2023-03-24 01:56:17,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dbaca42bc1724edabda106e24e191c2f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/dbaca42bc1724edabda106e24e191c2f 2023-03-24 01:56:17,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/fddc90d5449648f3b6205cd85ec31e1f 2023-03-24 01:56:17,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/69e6c627bbff44faa20e5696ad128d0a 2023-03-24 01:56:17,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1d055d43efb64785a65679a21eec943d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1d055d43efb64785a65679a21eec943d 2023-03-24 01:56:17,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/d85c0e06a4f44bbab54f15c70134eae8 2023-03-24 01:56:17,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/e4fd74d2436746fe93eaffdf572f7d76 2023-03-24 01:56:17,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/1cd8b82fe08344d3995b367e1dbab5bd 2023-03-24 01:56:17,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/recovered.edits/776.seqid, newMaxSeqId=776, maxSeqId=1 2023-03-24 01:56:17,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589. 2023-03-24 01:56:17,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7db42890c5a058f4b49bb0a2fb398589: 2023-03-24 01:56:17,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:17,149 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=7db42890c5a058f4b49bb0a2fb398589, regionState=CLOSED 2023-03-24 01:56:17,149 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622977149"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622977149"}]},"ts":"1679622977149"} 2023-03-24 01:56:17,151 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-24 01:56:17,151 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 7db42890c5a058f4b49bb0a2fb398589, server=jenkins-hbase4.apache.org,44439,1679622817494 in 2.1780 sec 2023-03-24 01:56:17,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-24 01:56:17,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7db42890c5a058f4b49bb0a2fb398589, UNASSIGN in 2.1830 sec 2023-03-24 01:56:17,153 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622977153"}]},"ts":"1679622977153"} 2023-03-24 01:56:17,153 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:56:17,155 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:56:17,156 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1950 sec 2023-03-24 01:56:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 01:56:19,070 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-24 01:56:19,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:56:19,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,073 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 01:56:19,073 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,079 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:19,081 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/recovered.edits] 2023-03-24 01:56:19,083 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/5f52e584cde74140b4de7a0e19d2f124 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/5f52e584cde74140b4de7a0e19d2f124 2023-03-24 01:56:19,085 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a0c343f169b649d59ff4d6a796096ace to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/a0c343f169b649d59ff4d6a796096ace 2023-03-24 01:56:19,086 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/bb5bcc6f270349ffbaa81ce5aab81257 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/A/bb5bcc6f270349ffbaa81ce5aab81257 2023-03-24 01:56:19,088 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/49470ac04e2c4829a68dfbfa28f75929 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/49470ac04e2c4829a68dfbfa28f75929 2023-03-24 01:56:19,089 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/658bf0329b7049cc9244f1e4319ed15a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/658bf0329b7049cc9244f1e4319ed15a 2023-03-24 01:56:19,090 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/8fc26a7828844fcea2589b35eec89573 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/B/8fc26a7828844fcea2589b35eec89573 2023-03-24 01:56:19,094 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/03ba016bdbee44bab0c27480ba693b06 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/03ba016bdbee44bab0c27480ba693b06 2023-03-24 01:56:19,095 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/087cf1ca4e2d4533b3c72de9ed55dd17 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/087cf1ca4e2d4533b3c72de9ed55dd17 2023-03-24 01:56:19,096 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a5866780e07c4fb6a9d993cf0a0689ea to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/C/a5866780e07c4fb6a9d993cf0a0689ea 2023-03-24 01:56:19,099 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/recovered.edits/776.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589/recovered.edits/776.seqid 2023-03-24 01:56:19,100 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/7db42890c5a058f4b49bb0a2fb398589 2023-03-24 01:56:19,100 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:56:19,101 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,103 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:56:19,104 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:56:19,105 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,105 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:56:19,105 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679622979105"}]},"ts":"9223372036854775807"} 2023-03-24 01:56:19,106 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:56:19,106 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7db42890c5a058f4b49bb0a2fb398589, NAME => 'TestAcidGuarantees,,1679622952105.7db42890c5a058f4b49bb0a2fb398589.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:56:19,106 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:56:19,106 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679622979106"}]},"ts":"9223372036854775807"} 2023-03-24 01:56:19,107 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:56:19,110 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 39 msec 2023-03-24 01:56:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 01:56:19,174 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-24 01:56:19,189 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=588 (was 509) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=446 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 171), AvailableMemoryMB=5770 (was 5370) - AvailableMemoryMB LEAK? - 2023-03-24 01:56:19,200 INFO [Listener at localhost/44783] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=588, MaxFileDescriptor=60000, SystemLoadAverage=446, ProcessCount=169, AvailableMemoryMB=5769 2023-03-24 01:56:19,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:56:19,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:56:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:19,205 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 01:56:19,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-24 01:56:19,205 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 01:56:19,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:19,206 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:19,206 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 empty. 2023-03-24 01:56:19,207 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:19,207 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:56:19,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:19,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:19,614 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:56:19,615 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4c6892182a9cd7f5a6bb4c2432955ea0, NAME => 'TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp 2023-03-24 01:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4c6892182a9cd7f5a6bb4c2432955ea0, disabling compactions & flushes 2023-03-24 01:56:20,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. after waiting 0 ms 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,026 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,026 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:20,028 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 01:56:20,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622980029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622980029"}]},"ts":"1679622980029"} 2023-03-24 01:56:20,030 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 01:56:20,030 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 01:56:20,031 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622980030"}]},"ts":"1679622980030"} 2023-03-24 01:56:20,031 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 01:56:20,034 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, ASSIGN}] 2023-03-24 01:56:20,036 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, ASSIGN 2023-03-24 01:56:20,036 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=false 2023-03-24 01:56:20,187 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:20,188 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622980187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622980187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622980187"}]},"ts":"1679622980187"} 2023-03-24 01:56:20,189 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:56:20,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:20,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c6892182a9cd7f5a6bb4c2432955ea0, NAME => 'TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:56:20,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:56:20,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,346 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,348 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,349 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A 2023-03-24 01:56:20,349 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A 2023-03-24 01:56:20,349 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:20,349 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName A 2023-03-24 01:56:20,350 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:20,350 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,351 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B 2023-03-24 01:56:20,351 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B 2023-03-24 01:56:20,351 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:20,351 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName B 2023-03-24 01:56:20,352 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:20,352 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,353 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C 2023-03-24 01:56:20,353 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C 2023-03-24 01:56:20,353 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:20,353 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName C 2023-03-24 01:56:20,353 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:20,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:56:20,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:20,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 01:56:20,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4c6892182a9cd7f5a6bb4c2432955ea0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61496972, jitterRate=-0.08362370729446411}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:56:20,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:20,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., pid=56, masterSystemTime=1679622980340 2023-03-24 01:56:20,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:20,361 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:20,361 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622980361"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622980361"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622980361"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622980361"}]},"ts":"1679622980361"} 2023-03-24 01:56:20,362 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-24 01:56:20,362 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 in 173 msec 2023-03-24 01:56:20,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-24 01:56:20,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, ASSIGN in 328 msec 2023-03-24 01:56:20,364 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 01:56:20,364 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679622980364"}]},"ts":"1679622980364"} 2023-03-24 01:56:20,365 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 01:56:20,366 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 01:56:20,367 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1640 sec 2023-03-24 01:56:21,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 01:56:21,309 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-24 01:56:21,310 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x3bafb41f to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:21,315 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68067de1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:21,316 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:21,317 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49458, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:21,318 DEBUG [Listener at localhost/44783] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 01:56:21,320 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 01:56:21,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 01:56:21,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 01:56:21,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:21,736 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:56:21,738 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 01:56:21,738 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 01:56:21,739 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 01:56:21,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, REOPEN/MOVE}] 2023-03-24 01:56:21,742 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, REOPEN/MOVE 2023-03-24 01:56:21,743 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:21,743 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622981743"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622981743"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622981743"}]},"ts":"1679622981743"} 2023-03-24 01:56:21,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:56:21,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:21,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4c6892182a9cd7f5a6bb4c2432955ea0, disabling compactions & flushes 2023-03-24 01:56:21,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:21,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:21,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. after waiting 0 ms 2023-03-24 01:56:21,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:21,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 01:56:21,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:21,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:21,900 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4c6892182a9cd7f5a6bb4c2432955ea0 to self. 2023-03-24 01:56:21,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:21,902 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=CLOSED 2023-03-24 01:56:21,902 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622981902"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679622981902"}]},"ts":"1679622981902"} 2023-03-24 01:56:21,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-24 01:56:21,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 in 159 msec 2023-03-24 01:56:21,904 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,44439,1679622817494; forceNewPlan=false, retain=true 2023-03-24 01:56:22,055 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,055 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622982055"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679622982055"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679622982055"}]},"ts":"1679622982055"} 2023-03-24 01:56:22,056 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:56:22,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c6892182a9cd7f5a6bb4c2432955ea0, NAME => 'TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.', STARTKEY => '', ENDKEY => ''} 2023-03-24 01:56:22,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 01:56:22,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,211 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,212 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A 2023-03-24 01:56:22,212 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A 2023-03-24 01:56:22,212 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:22,213 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName A 2023-03-24 01:56:22,214 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:22,214 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,214 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B 2023-03-24 01:56:22,214 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B 2023-03-24 01:56:22,215 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:22,215 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName B 2023-03-24 01:56:22,215 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:22,215 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,216 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C 2023-03-24 01:56:22,216 DEBUG [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C 2023-03-24 01:56:22,216 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 01:56:22,216 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c6892182a9cd7f5a6bb4c2432955ea0 columnFamilyName C 2023-03-24 01:56:22,216 INFO [StoreOpener-4c6892182a9cd7f5a6bb4c2432955ea0-1] regionserver.HStore(310): Store=4c6892182a9cd7f5a6bb4c2432955ea0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 01:56:22,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 01:56:22,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4c6892182a9cd7f5a6bb4c2432955ea0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61212574, jitterRate=-0.087861567735672}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 01:56:22,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:22,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., pid=61, masterSystemTime=1679622982207 2023-03-24 01:56:22,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,223 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,223 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679622982223"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679622982223"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679622982223"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679622982223"}]},"ts":"1679622982223"} 2023-03-24 01:56:22,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-24 01:56:22,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 in 168 msec 2023-03-24 01:56:22,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-24 01:56:22,226 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, REOPEN/MOVE in 483 msec 2023-03-24 01:56:22,228 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-24 01:56:22,228 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-03-24 01:56:22,229 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-03-24 01:56:22,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-24 01:56:22,230 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,235 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,235 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,239 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,239 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,243 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,243 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,248 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,248 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,252 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,252 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,257 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,257 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,263 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,263 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,271 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,271 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,278 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,279 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:62686 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 01:56:22,283 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 01:56:22,293 DEBUG [hconnection-0x7e4ef054-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,295 DEBUG [hconnection-0x2376f5c7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,295 DEBUG [hconnection-0x3bf03c53-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,295 DEBUG [hconnection-0x684744b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,295 DEBUG [hconnection-0x28066334-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,296 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,296 DEBUG [hconnection-0x2627dfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,301 DEBUG [hconnection-0x694ffec5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,301 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49478, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,303 DEBUG [hconnection-0x13fbea4d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,303 DEBUG [hconnection-0x4164cebe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,303 DEBUG [hconnection-0x6d78a6ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 01:56:22,307 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49502, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,305 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,305 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49472, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,309 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,310 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,310 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49524, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,317 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49510, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,318 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49532, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 01:56:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,320 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,323 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,325 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,325 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:22,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:56:22,327 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:22,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:22,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:22,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,333 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c8573b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,337 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,338 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,338 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,342 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,343 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623042344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623042345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622982338, max timestamp=1679622982345Num uniques 859; 2023-03-24 01:56:22,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 01:56:22,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622982338, max timestamp=1679622982345Num uniques 859; 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622982338, max timestamp=1679622982345Num uniques 859; 2023-03-24 01:56:22,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:22,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 01:56:22,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623042345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,350 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,353 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,353 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623042369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623042370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623042448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623042450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,453 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:22,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:22,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623042451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,457 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,457 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,458 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:22,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,460 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,460 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,462 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d4b755a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,465 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,468 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,470 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,470 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,470 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,472 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,474 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623042473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623042473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,476 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,477 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:22,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,483 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,483 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,584 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:22,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:22,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,587 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,587 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,588 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,589 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,590 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:22,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,592 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,595 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c9191[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,597 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,598 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,599 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,600 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,603 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,603 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,604 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,604 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,604 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,605 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,606 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,610 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,610 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,610 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,610 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623042651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623042653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623042655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,664 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:56:22,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623042676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623042677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,710 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:22,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:22,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,716 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,716 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,719 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,719 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,720 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,724 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,724 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,724 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,724 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,729 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,729 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,730 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2572f4e9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,733 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,733 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,734 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,734 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,735 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,738 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,738 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,741 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,746 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,746 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,747 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:22,746 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,751 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324929b1dc52de64de1ba7a846640078185 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324929b1dc52de64de1ba7a846640078185 2023-03-24 01:56:22,847 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:22,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:22,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,850 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,850 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,851 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,852 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:22,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,856 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,856 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,856 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,856 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,858 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,858 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,858 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,858 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25031ef3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,862 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,863 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,866 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,866 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,866 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,868 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,872 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,873 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,873 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,873 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:22,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623042953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623042957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623042958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,973 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:22,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:22,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:22,976 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:22,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:22,976 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:22,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,977 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623042977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:22,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623042979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,979 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,979 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,980 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:22,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:22,981 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:22,982 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:22,983 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,983 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73b44eac[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:22,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:22,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,986 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:22,986 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:22,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:22,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:22,988 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,988 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:22,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:22,991 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:22,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:22,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:22,991 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:22,991 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:22,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:22,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:22,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:22,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:22,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:22,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:22,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:22,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:22,997 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:22,998 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:22,998 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:22,998 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:22,998 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,998 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:22,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:22,998 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,098 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,101 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,101 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,102 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,103 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,103 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,104 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,105 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,105 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,107 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,107 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c1d952[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,109 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,114 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,118 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,118 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,118 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,121 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,122 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,124 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,124 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,124 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,156 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ab747550bef84a44886303fe3f6a6104 2023-03-24 01:56:23,225 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,228 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,228 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,231 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,231 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,232 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,232 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,233 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,233 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,233 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,233 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,235 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2426b2df[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,236 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,237 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,238 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,238 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,239 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,243 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,243 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,245 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,246 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,249 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,249 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,249 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,249 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,349 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,353 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,353 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,355 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,355 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,356 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,357 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,357 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,358 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,358 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196fc42e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,362 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,363 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,363 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,364 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,364 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,367 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,367 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,367 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,373 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,373 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,373 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,374 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:23,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623043458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:23,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623043459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:23,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623043463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,474 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,477 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,478 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,480 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,480 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:23,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623043480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,482 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,482 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,482 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,482 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,483 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,483 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:23,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623043481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,484 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,484 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,485 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,485 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ff3881[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,486 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,488 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,489 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,489 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,490 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,493 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,493 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,493 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,493 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,493 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,504 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,507 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,507 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,507 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/964faaca01274b5895528d764a2981dd 2023-03-24 01:56:23,608 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,611 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,611 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,612 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,613 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,614 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,614 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,614 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,614 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,615 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,616 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ec6385[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,618 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,618 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,620 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,622 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,625 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,625 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,625 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,627 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,628 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,629 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,631 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,632 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,633 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,633 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,733 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,736 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,736 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,738 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,738 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,739 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,739 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,740 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,740 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,741 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,741 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414d830d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,743 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,744 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,744 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,744 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,744 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,744 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,744 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,747 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,747 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,750 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,750 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,751 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,751 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,751 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,752 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,753 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,755 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,756 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,756 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,756 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,756 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,856 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,859 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,860 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,861 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,861 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,862 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,863 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,865 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e461be[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,867 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:23,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:23,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:23,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:23,873 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:23,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:23,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:23,873 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:23,873 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:23,875 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,875 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,875 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,876 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,877 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:23,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:23,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:23,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,884 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:23,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,884 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:23,884 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:23,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:23,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/e6cd4cc03a084bdcbefbcfaaefa10baf 2023-03-24 01:56:23,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ab747550bef84a44886303fe3f6a6104 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104 2023-03-24 01:56:23,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104, entries=150, sequenceid=15, filesize=25.1 K 2023-03-24 01:56:23,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/964faaca01274b5895528d764a2981dd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd 2023-03-24 01:56:23,984 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:23,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:23,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 01:56:23,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/e6cd4cc03a084bdcbefbcfaaefa10baf as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf 2023-03-24 01:56:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:23,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:23,988 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:23,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:23,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:23,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,990 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:23,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:23,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,990 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:23,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 01:56:23,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:23,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:23,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:23,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:23,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 1665ms, sequenceid=15, compaction requested=false 2023-03-24 01:56:23,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:23,992 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 01:56:23,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:23,992 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:23,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:23,993 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,993 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:23,994 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,994 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ee6ed3[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:23,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:23,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:23,996 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:23,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:23,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:24,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:24,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:24,001 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,001 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:24,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:24,004 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:24,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:24,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:24,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:24,004 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:24,006 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:24,006 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:24,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,006 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,008 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,010 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:24,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:24,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:24,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:24,010 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,011 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:24,011 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:24,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:24,111 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:24,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:24,114 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:24,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:24,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:24,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:24,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:24,116 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:24,116 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:24,116 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,116 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:24,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:24,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:24,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:24,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:24,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:24,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:24,118 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=126.28 KB heapSize=362.32 KB 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:24,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:24,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:24,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:24,132 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324524e1c4d9d8346c8a5e24d4c4dc5d681 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324524e1c4d9d8346c8a5e24d4c4dc5d681 2023-03-24 01:56:24,137 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=42.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b9aaf16761a247c1a03000f8efa77e65 2023-03-24 01:56:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:24,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:24,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623044477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623044477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623044479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623044485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623044485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,553 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/24f99e5481c9484384b2b31429b6a885 2023-03-24 01:56:24,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623044580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623044580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623044583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623044783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623044784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:24,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623044787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,964 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/5c0e2198c8b34a5e922fe74935e6189f 2023-03-24 01:56:24,968 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b9aaf16761a247c1a03000f8efa77e65 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65 2023-03-24 01:56:24,971 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65, entries=150, sequenceid=51, filesize=25.1 K 2023-03-24 01:56:24,972 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/24f99e5481c9484384b2b31429b6a885 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885 2023-03-24 01:56:24,975 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 01:56:24,975 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/5c0e2198c8b34a5e922fe74935e6189f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f 2023-03-24 01:56:24,979 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f, entries=150, sequenceid=51, filesize=11.6 K 2023-03-24 01:56:24,980 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.28 KB/129315, heapSize ~362.27 KB/370968, currentSize=53.67 KB/54960 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 862ms, sequenceid=51, compaction requested=false 2023-03-24 01:56:24,981 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:24,981 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:24,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:24,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:24,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:24,981 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:24,984 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:24,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24add7e0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:24,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:24,986 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:24,987 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,988 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:24,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,000 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,000 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,000 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,003 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,004 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,008 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,008 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,009 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,009 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:25,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:25,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:25,106 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032496a758f2677946ab85479c579cc1ecde to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032496a758f2677946ab85479c579cc1ecde 2023-03-24 01:56:25,109 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,112 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ec8c5f421b6f4eb5bef38c278b181fb0 2023-03-24 01:56:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,113 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,113 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,114 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,115 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,116 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,116 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,118 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623045121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623045123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,126 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623045124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9557d93[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,127 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,128 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,129 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,129 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,130 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,132 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,135 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,135 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,137 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,139 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,144 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,144 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,144 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623045225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623045226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623045228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,245 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,248 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,250 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,251 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,251 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,252 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,252 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,253 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,253 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,257 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,258 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb90496[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,261 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,262 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,263 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,263 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,265 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,265 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,265 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,266 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,267 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,270 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,271 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,271 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,371 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,374 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,374 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,375 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,376 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,377 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,379 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68456b24[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,380 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,382 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,382 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,386 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,386 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,386 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,388 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,389 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,390 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,390 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,391 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,391 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,391 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:25,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623045429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623045429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623045431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,491 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,495 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,495 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,497 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,498 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,499 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,499 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,499 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,499 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,499 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,501 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,501 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229da31b[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,504 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,504 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,504 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,506 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,508 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,508 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,511 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,511 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,513 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,515 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,515 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,516 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,516 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/30d10eae645e40bdb4c0f6de29708e15 2023-03-24 01:56:25,616 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,619 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,619 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,619 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,619 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,619 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,619 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,621 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,621 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,622 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,623 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,623 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,623 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,623 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,623 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,629 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86eb3e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,630 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,631 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,632 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,632 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,632 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,633 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,633 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,635 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,636 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,636 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,637 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,637 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,637 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,639 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,640 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,644 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,644 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,644 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,644 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623045732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623045733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623045735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,745 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,748 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,748 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,748 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,748 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,748 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,749 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,750 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,750 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,751 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,751 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,751 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,752 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,752 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,753 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,753 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44dfc20c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,755 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,756 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,756 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,756 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,758 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,758 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,761 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,761 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,761 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,762 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,763 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,764 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,766 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,766 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,766 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,767 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,767 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,866 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,870 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,870 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,872 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,872 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,872 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,872 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:25,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:25,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:25,874 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,874 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:25,875 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:25,875 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:25,875 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:25,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:25,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7808df8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:25,879 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:25,879 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:25,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,880 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:25,880 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:25,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,882 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:25,884 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:25,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:25,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:25,884 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:25,884 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:25,885 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,885 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,885 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:25,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:25,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,887 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:25,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:25,891 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:25,891 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:25,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:25,890 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:25,890 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:25,891 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:25,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:25,942 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/22b3fdea79a3489187c2d004a797a445 2023-03-24 01:56:25,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ec8c5f421b6f4eb5bef38c278b181fb0 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0 2023-03-24 01:56:25,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0, entries=150, sequenceid=63, filesize=25.1 K 2023-03-24 01:56:25,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/30d10eae645e40bdb4c0f6de29708e15 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15 2023-03-24 01:56:25,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15, entries=100, sequenceid=63, filesize=9.3 K 2023-03-24 01:56:25,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/22b3fdea79a3489187c2d004a797a445 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445 2023-03-24 01:56:25,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445, entries=100, sequenceid=63, filesize=9.3 K 2023-03-24 01:56:25,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 866ms, sequenceid=63, compaction requested=true 2023-03-24 01:56:25,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:25,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:25,959 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:25,959 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:25,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:25,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:25,960 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77234 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:25,961 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:25,961 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:25,961 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,961 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:25,961 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=75.4 K 2023-03-24 01:56:25,961 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:25,961 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=32.5 K 2023-03-24 01:56:25,961 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ab747550bef84a44886303fe3f6a6104, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622982317 2023-03-24 01:56:25,964 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 964faaca01274b5895528d764a2981dd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622982317 2023-03-24 01:56:25,964 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b9aaf16761a247c1a03000f8efa77e65, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622982364 2023-03-24 01:56:25,964 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 24f99e5481c9484384b2b31429b6a885, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622982364 2023-03-24 01:56:25,964 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting ec8c5f421b6f4eb5bef38c278b181fb0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622984476 2023-03-24 01:56:25,965 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 30d10eae645e40bdb4c0f6de29708e15, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622984476 2023-03-24 01:56:25,971 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:25,981 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#522 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:25,991 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:25,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:25,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:25,996 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:25,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:25,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:25,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:25,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,998 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:25,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:25,998 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:25,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:25,999 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:25,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:26,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:26,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:26,001 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:26,001 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,019 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032479906b097e504675b815523f0b5de022 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032479906b097e504675b815523f0b5de022 2023-03-24 01:56:26,023 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/30f75c8e4299403a9b59c60c71448159 2023-03-24 01:56:26,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:26,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:26,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623046268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623046273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623046273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623046374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623046378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623046378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,391 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/84493a50cb7b4d93b47049f0b84741b6 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/84493a50cb7b4d93b47049f0b84741b6 2023-03-24 01:56:26,395 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 84493a50cb7b4d93b47049f0b84741b6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:26,395 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:26,395 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=13, startTime=1679622985959; duration=0sec 2023-03-24 01:56:26,395 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:26,395 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:26,396 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:26,396 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:26,396 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:26,396 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=32.5 K 2023-03-24 01:56:26,397 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e6cd4cc03a084bdcbefbcfaaefa10baf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679622982317 2023-03-24 01:56:26,397 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5c0e2198c8b34a5e922fe74935e6189f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679622982364 2023-03-24 01:56:26,397 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 22b3fdea79a3489187c2d004a797a445, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622984476 2023-03-24 01:56:26,410 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:26,434 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/eda34b35dd794039b1ff3f7cafbf6bc3 2023-03-24 01:56:26,438 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/3e4069b3aee4442b9adcbca452d93f30 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3e4069b3aee4442b9adcbca452d93f30 2023-03-24 01:56:26,446 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 3e4069b3aee4442b9adcbca452d93f30(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:26,446 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:26,446 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=13, startTime=1679622985959; duration=0sec 2023-03-24 01:56:26,446 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:26,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623046498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,502 DEBUG [Thread-3892] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:26,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623046506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,510 DEBUG [Thread-3894] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:26,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623046578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623046581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623046582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,833 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/f6706e29504847d9bf26525ca829927d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f6706e29504847d9bf26525ca829927d 2023-03-24 01:56:26,837 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into f6706e29504847d9bf26525ca829927d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:26,837 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:26,837 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=13, startTime=1679622985960; duration=0sec 2023-03-24 01:56:26,837 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:26,849 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/99cf7e837a2e4297b42707eab38a1d08 2023-03-24 01:56:26,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/30f75c8e4299403a9b59c60c71448159 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159 2023-03-24 01:56:26,856 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159, entries=150, sequenceid=83, filesize=25.1 K 2023-03-24 01:56:26,857 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/eda34b35dd794039b1ff3f7cafbf6bc3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3 2023-03-24 01:56:26,860 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3, entries=150, sequenceid=83, filesize=11.6 K 2023-03-24 01:56:26,861 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/99cf7e837a2e4297b42707eab38a1d08 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08 2023-03-24 01:56:26,864 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08, entries=150, sequenceid=83, filesize=11.6 K 2023-03-24 01:56:26,865 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 864ms, sequenceid=83, compaction requested=false 2023-03-24 01:56:26,865 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:26,865 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:26,869 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,869 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:26,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:26,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484a6368[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:26,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:26,872 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,876 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:26,877 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,877 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:26,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:26,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:26,878 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,878 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:26,881 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:26,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:26,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:26,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:26,881 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:26,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:26,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 01:56:26,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:26,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:26,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:26,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:26,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:26,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:26,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:26,883 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,888 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:26,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:26,889 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:26,889 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:26,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:26,889 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:26,889 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:26,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623046904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623046905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623046905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,989 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:26,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:26,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:26,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:26,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:26,993 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:26,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:26,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:26,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:26,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,993 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,995 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,995 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:26,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:26,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:26,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:26,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:26,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:26,997 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:26,997 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:26,997 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:26,997 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:26,998 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:26,999 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:26,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:26,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:26,999 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa2be44[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:27,001 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:27,001 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,002 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:27,002 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:27,002 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,002 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:27,003 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,003 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:27,005 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:27,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:27,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:27,005 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:27,005 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:27,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,008 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623047008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623047008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:27,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:27,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:27,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,011 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:27,011 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623047008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,112 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:27,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:27,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:27,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:27,115 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,115 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,117 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,117 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:27,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:27,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:27,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,118 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:27,119 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:27,119 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:27,120 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4557cc1e[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:27,123 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:27,123 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:27,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:27,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:27,125 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,125 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:27,126 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,126 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:27,129 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:27,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:27,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:27,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:27,129 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:27,131 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,131 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,131 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,131 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,132 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:27,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,135 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:27,135 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623047213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623047213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623047213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,236 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:27,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:27,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:27,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:27,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:27,239 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:27,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:27,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:27,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,239 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,241 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,241 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:27,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:27,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:27,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:27,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:27,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:27,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:27,243 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,244 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:27,244 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:27,244 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:27,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:27,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463b7601[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:27,247 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:27,247 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,248 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:27,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,248 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:27,253 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,253 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:27,256 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:27,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:27,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:27,256 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:27,256 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:27,259 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,259 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:27,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,259 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,261 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:27,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:27,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:27,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:27,263 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:27,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:27,264 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:27,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:27,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,264 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:27,296 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032449e92aaa72f245099ac3277ed64e0ad4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032449e92aaa72f245099ac3277ed64e0ad4 2023-03-24 01:56:27,301 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/912cd5bd5e6646b6a348826371941245 2023-03-24 01:56:27,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/1914d69be39344f4823aa24dedd7739e 2023-03-24 01:56:27,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/72250d8a15fa4751ae95c9535affa6ac 2023-03-24 01:56:27,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/912cd5bd5e6646b6a348826371941245 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245 2023-03-24 01:56:27,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245, entries=200, sequenceid=100, filesize=31.9 K 2023-03-24 01:56:27,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/1914d69be39344f4823aa24dedd7739e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e 2023-03-24 01:56:27,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e, entries=150, sequenceid=100, filesize=11.6 K 2023-03-24 01:56:27,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/72250d8a15fa4751ae95c9535affa6ac as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac 2023-03-24 01:56:27,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac, entries=150, sequenceid=100, filesize=11.6 K 2023-03-24 01:56:27,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 455ms, sequenceid=100, compaction requested=true 2023-03-24 01:56:27,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:27,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:27,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:27,338 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:27,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:27,338 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:27,339 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:27,339 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:27,339 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,339 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3e4069b3aee4442b9adcbca452d93f30, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=82.3 K 2023-03-24 01:56:27,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:27,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:27,340 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,340 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/84493a50cb7b4d93b47049f0b84741b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=34.9 K 2023-03-24 01:56:27,340 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 3e4069b3aee4442b9adcbca452d93f30, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622982366 2023-03-24 01:56:27,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 84493a50cb7b4d93b47049f0b84741b6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622982366 2023-03-24 01:56:27,341 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 30f75c8e4299403a9b59c60c71448159, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679622985120 2023-03-24 01:56:27,341 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting eda34b35dd794039b1ff3f7cafbf6bc3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679622985120 2023-03-24 01:56:27,341 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 912cd5bd5e6646b6a348826371941245, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986250 2023-03-24 01:56:27,341 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1914d69be39344f4823aa24dedd7739e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986271 2023-03-24 01:56:27,350 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:27,353 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#532 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:27,364 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:27,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:27,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:27,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:27,368 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:27,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:27,368 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:27,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,369 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,370 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:27,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:27,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:27,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:27,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:27,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:27,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:27,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:27,372 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:27,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:27,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:27,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:27,373 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:27,373 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,373 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:27,373 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:56:27,373 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:27,373 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:27,373 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:27,374 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:27,374 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:27,374 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:27,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:27,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:27,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623047552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623047552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623047552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623047655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623047655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623047656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,712 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 01:56:27,769 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/c3de809a300c40678cb2ee5057b759c5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c3de809a300c40678cb2ee5057b759c5 2023-03-24 01:56:27,783 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into c3de809a300c40678cb2ee5057b759c5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:27,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:27,783 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=13, startTime=1679622987338; duration=0sec 2023-03-24 01:56:27,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:27,783 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:27,784 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:27,784 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:27,784 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:27,784 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f6706e29504847d9bf26525ca829927d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=34.9 K 2023-03-24 01:56:27,785 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f6706e29504847d9bf26525ca829927d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679622982366 2023-03-24 01:56:27,785 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 99cf7e837a2e4297b42707eab38a1d08, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679622985120 2023-03-24 01:56:27,785 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 72250d8a15fa4751ae95c9535affa6ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986271 2023-03-24 01:56:27,790 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b6b970c258df48d585e8b0b3c458b12e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b6b970c258df48d585e8b0b3c458b12e 2023-03-24 01:56:27,794 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:27,809 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/5d04ff6ac26d447f8eab4152de186e67 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5d04ff6ac26d447f8eab4152de186e67 2023-03-24 01:56:27,813 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 5d04ff6ac26d447f8eab4152de186e67(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:27,813 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:27,813 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=13, startTime=1679622987338; duration=0sec 2023-03-24 01:56:27,813 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:27,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623047858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623047858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:27,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:27,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623047859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623048161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623048161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623048163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,204 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/671dafcd53a5498f8afb022d3f85db5e 2023-03-24 01:56:28,209 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/9f2a3186a54c4b8eaaafc6b906bced90 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/9f2a3186a54c4b8eaaafc6b906bced90 2023-03-24 01:56:28,213 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 9f2a3186a54c4b8eaaafc6b906bced90(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:28,213 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:28,213 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=13, startTime=1679622987338; duration=0sec 2023-03-24 01:56:28,214 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:28,218 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6b7a15e9331548a889e85f01e82c7136 2023-03-24 01:56:28,227 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/801f0bd4b018436a8c687c6cc277b7a5 2023-03-24 01:56:28,230 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/671dafcd53a5498f8afb022d3f85db5e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e 2023-03-24 01:56:28,233 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e, entries=150, sequenceid=119, filesize=25.1 K 2023-03-24 01:56:28,234 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6b7a15e9331548a889e85f01e82c7136 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136 2023-03-24 01:56:28,237 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136, entries=150, sequenceid=119, filesize=11.6 K 2023-03-24 01:56:28,238 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/801f0bd4b018436a8c687c6cc277b7a5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5 2023-03-24 01:56:28,242 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5, entries=150, sequenceid=119, filesize=11.6 K 2023-03-24 01:56:28,242 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 869ms, sequenceid=119, compaction requested=false 2023-03-24 01:56:28,242 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:28,242 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:28,242 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:28,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:28,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:28,243 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:28,245 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,245 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:28,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df10c8d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:28,249 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:28,249 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,250 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:28,250 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:28,251 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,251 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:28,252 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:28,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:28,255 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:28,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:28,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:28,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:28,255 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:28,257 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,257 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,257 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:28,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,259 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,260 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,263 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:28,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:28,264 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:28,264 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:28,264 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:28,264 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:28,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:28,364 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:28,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:28,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:28,370 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:28,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:28,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:28,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:28,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,370 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,371 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:28,371 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:28,372 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,372 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:28,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:28,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:28,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:28,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:28,378 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:28,378 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:28,378 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:28,378 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 01:56:28,378 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:28,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:28,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:28,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,393 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ac3a5eec80204c56822f12e5ed0745cd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac3a5eec80204c56822f12e5ed0745cd 2023-03-24 01:56:28,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:28,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:28,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623048680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623048681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623048681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623048783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623048785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623048785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,802 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1ebdd4d0b31c43628bb4fb057c428f8e 2023-03-24 01:56:28,811 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6618d6be9fe14ac2b59a5b1aac6269ae 2023-03-24 01:56:28,819 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/2d137cb35e3e4c08a2cee45a60443ef8 2023-03-24 01:56:28,822 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1ebdd4d0b31c43628bb4fb057c428f8e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e 2023-03-24 01:56:28,825 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e, entries=150, sequenceid=136, filesize=25.2 K 2023-03-24 01:56:28,825 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6618d6be9fe14ac2b59a5b1aac6269ae as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae 2023-03-24 01:56:28,829 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae, entries=150, sequenceid=136, filesize=11.7 K 2023-03-24 01:56:28,829 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/2d137cb35e3e4c08a2cee45a60443ef8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8 2023-03-24 01:56:28,833 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8, entries=150, sequenceid=136, filesize=11.7 K 2023-03-24 01:56:28,834 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 456ms, sequenceid=136, compaction requested=true 2023-03-24 01:56:28,834 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:28,834 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:28,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:28,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:28,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:28,834 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:28,836 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,837 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,837 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,837 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:28,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374bd9e9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:28,841 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:28,841 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,842 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:28,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:28,844 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:28,848 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:28,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:28,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:28,848 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:28,848 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:28,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,850 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:28,851 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,853 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:28,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:28,857 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:28,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:28,857 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:28,857 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:28,957 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:28,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:28,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:28,961 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:28,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:28,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:28,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:28,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,961 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,963 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:28,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:28,963 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:28,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:28,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:28,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:28,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:28,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:28,965 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:28,965 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:28,965 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:28,965 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:28,966 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:28,981 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324435ea0434d6644e0b89c321af68f8c74 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324435ea0434d6644e0b89c321af68f8c74 2023-03-24 01:56:28,987 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b4245094f87746deb43048b07593dff4 2023-03-24 01:56:28,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:28,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:29,001 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/c0ccb0189b534bdabf1bb757533be3f4 2023-03-24 01:56:29,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,409 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/1bf6d39f63964bdf8e2880cb06744c10 2023-03-24 01:56:29,412 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b4245094f87746deb43048b07593dff4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4 2023-03-24 01:56:29,415 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4, entries=150, sequenceid=154, filesize=25.3 K 2023-03-24 01:56:29,416 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/c0ccb0189b534bdabf1bb757533be3f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4 2023-03-24 01:56:29,419 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4, entries=150, sequenceid=154, filesize=11.8 K 2023-03-24 01:56:29,420 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/1bf6d39f63964bdf8e2880cb06744c10 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10 2023-03-24 01:56:29,425 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10, entries=150, sequenceid=154, filesize=11.8 K 2023-03-24 01:56:29,426 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 461ms, sequenceid=154, compaction requested=true 2023-03-24 01:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:29,428 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c07d422[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:29,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:29,429 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,431 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:29,431 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,431 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:29,433 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:29,435 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:29,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:29,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:29,435 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:29,435 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:29,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,437 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:29,438 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,440 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:29,445 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:29,445 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:29,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:29,446 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:29,446 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:29,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:29,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:29,546 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:29,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:29,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:29,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:29,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:29,550 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:29,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:29,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:29,550 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,551 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:29,558 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:29,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:29,558 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,559 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:29,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:29,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:29,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:29,560 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:29,560 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:29,560 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 01:56:29,560 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:29,561 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:29,561 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:29,561 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:29,561 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:29,561 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:29,572 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032461dc5fd142b4493ba49774fe80f8669d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461dc5fd142b4493ba49774fe80f8669d 2023-03-24 01:56:29,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:29,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:29,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623049942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623049943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623049944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:29,977 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b7d97d5bd2de443d81136c8fafe3ee42 2023-03-24 01:56:30,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623050248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623050249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623050251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,385 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/330c359a86124ffb9c4b239486cc2dbf 2023-03-24 01:56:30,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623050515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,519 DEBUG [Thread-3894] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:30,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623050533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,537 DEBUG [Thread-3892] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:30,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623050752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623050754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:30,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623050755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,795 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/f5af9df027f049dbbdb0fd82469b9577 2023-03-24 01:56:30,799 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/b7d97d5bd2de443d81136c8fafe3ee42 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42 2023-03-24 01:56:30,803 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42, entries=150, sequenceid=168, filesize=25.3 K 2023-03-24 01:56:30,804 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/330c359a86124ffb9c4b239486cc2dbf as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf 2023-03-24 01:56:30,807 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf, entries=150, sequenceid=168, filesize=11.8 K 2023-03-24 01:56:30,808 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/f5af9df027f049dbbdb0fd82469b9577 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577 2023-03-24 01:56:30,811 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577, entries=150, sequenceid=168, filesize=11.8 K 2023-03-24 01:56:30,812 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 1251ms, sequenceid=168, compaction requested=true 2023-03-24 01:56:30,812 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:30,812 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:30,813 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,813 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:30,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@511128dd[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:30,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:30,815 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,816 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:30,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:30,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:30,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:30,819 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:30,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:30,822 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:30,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:30,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:30,822 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:30,822 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:30,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,823 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:30,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,824 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:30,828 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:30,831 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:30,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:30,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:30,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:30,831 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:30,832 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,832 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:30,832 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:30,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:30,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:30,932 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:30,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:30,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:30,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:30,935 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:30,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:30,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:30,935 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:30,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,936 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:30,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,937 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:30,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:30,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:30,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:30,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:30,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:30,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:30,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:30,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:30,938 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:30,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:30,939 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:30,939 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:31,355 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032494b4c156525c4f2e9665f7501ab353d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032494b4c156525c4f2e9665f7501ab353d3 2023-03-24 01:56:31,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:31,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:31,760 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1d1d66ac2b8f4d429cafad56c4cd40aa 2023-03-24 01:56:31,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623051773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623051776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623051776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,786 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/8717fc4203fe46bc86c443140fb53f2c 2023-03-24 01:56:31,798 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/b115050252c34cbf8796bb121a1b99da 2023-03-24 01:56:31,801 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1d1d66ac2b8f4d429cafad56c4cd40aa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa 2023-03-24 01:56:31,805 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa, entries=150, sequenceid=187, filesize=25.3 K 2023-03-24 01:56:31,806 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/8717fc4203fe46bc86c443140fb53f2c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c 2023-03-24 01:56:31,809 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c, entries=150, sequenceid=187, filesize=11.8 K 2023-03-24 01:56:31,810 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/b115050252c34cbf8796bb121a1b99da as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da 2023-03-24 01:56:31,813 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da, entries=150, sequenceid=187, filesize=11.8 K 2023-03-24 01:56:31,814 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 875ms, sequenceid=187, compaction requested=true 2023-03-24 01:56:31,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:31,814 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:31,814 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:31,816 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,816 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,817 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1c316[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:31,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:31,818 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,819 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:31,820 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:31,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,820 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:31,821 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:31,821 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:31,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:31,825 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:31,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:31,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:31,825 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:31,825 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:31,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,827 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:31,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:31,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,828 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,829 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,831 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,834 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:31,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:31,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:31,834 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:31,834 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:31,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:31,834 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,835 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:31,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:31,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:31,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:31,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:31,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:31,892 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ed88493264674154ad38a2921e92b7fd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ed88493264674154ad38a2921e92b7fd 2023-03-24 01:56:31,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623051905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623051905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:31,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623051906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,935 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:31,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:31,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:31,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:31,938 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:31,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:31,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:31,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:31,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,940 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:31,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:31,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,940 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:31,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:31,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:31,941 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:31,941 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:31,941 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:31,942 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:31,942 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:31,943 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,943 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:31,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d3cc6f4[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:31,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:31,945 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,946 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:31,946 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:31,949 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:31,949 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:31,951 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:31,951 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:31,951 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:31,951 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:31,951 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:31,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:31,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:31,954 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:31,954 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,955 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,958 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:31,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:31,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:31,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:31,958 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:31,959 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:31,958 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:31,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:31,959 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623052012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623052012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623052012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,059 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:32,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:32,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:32,063 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:32,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:32,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:32,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,063 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,065 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,065 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,066 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:32,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:32,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:32,067 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:32,067 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,067 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:32,067 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:32,068 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:32,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:32,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:32,068 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:32,069 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,069 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,070 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:32,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe8b9b5[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:32,071 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:32,071 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,072 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:32,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:32,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:32,077 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:32,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:32,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:32,077 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:32,077 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:32,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,079 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,080 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,081 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,083 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:32,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:32,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,084 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,084 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:32,084 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:32,084 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,184 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:32,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:32,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:32,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:32,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:32,188 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:32,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:32,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:32,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:32,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:32,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:32,191 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:32,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:32,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:32,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:32,192 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,192 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:32,192 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:32,192 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,192 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:32,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:32,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:32,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:32,195 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,195 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:32,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e73d55[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:32,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:32,196 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,198 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:32,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:32,198 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:32,198 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,199 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:32,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,200 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,201 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:32,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:32,203 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:32,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:32,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:32,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:32,203 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:32,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,207 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,211 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:32,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,211 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:32,211 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,211 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,212 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623052216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623052216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49510 deadline: 1679623052216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,296 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=202, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/0be12a6d5466401d968c995480be522f 2023-03-24 01:56:32,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/cac2cf3be3f9456aac3ae7893768e576 2023-03-24 01:56:32,311 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:32,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:32,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:32,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:32,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:32,316 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:32,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:32,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:32,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,316 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,319 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,319 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,320 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:32,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:32,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:32,328 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:32,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,328 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:32,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:32,329 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:32,329 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:32,330 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,330 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,331 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d249fcc[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:32,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:32,333 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/46fcd7c75b434ed1abc7e202e7c42a03 2023-03-24 01:56:32,335 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,335 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:32,335 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:32,337 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:32,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:32,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:32,341 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:32,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/0be12a6d5466401d968c995480be522f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f 2023-03-24 01:56:32,344 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,345 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f, entries=150, sequenceid=202, filesize=25.3 K 2023-03-24 01:56:32,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/cac2cf3be3f9456aac3ae7893768e576 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576 2023-03-24 01:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:32,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:32,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,352 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,352 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:32,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,352 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:32,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576, entries=150, sequenceid=202, filesize=11.8 K 2023-03-24 01:56:32,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/46fcd7c75b434ed1abc7e202e7c42a03 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03 2023-03-24 01:56:32,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03, entries=150, sequenceid=202, filesize=11.8 K 2023-03-24 01:56:32,361 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 483ms, sequenceid=202, compaction requested=true 2023-03-24 01:56:32,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:32,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:32,362 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 01:56:32,362 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 01:56:32,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:32,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:32,365 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 181128 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 01:56:32,365 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:32,365 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,365 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/9f2a3186a54c4b8eaaafc6b906bced90, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=176.9 K 2023-03-24 01:56:32,365 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84108 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 01:56:32,365 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:32,365 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,365 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c3de809a300c40678cb2ee5057b759c5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=82.1 K 2023-03-24 01:56:32,366 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c3de809a300c40678cb2ee5057b759c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986271 2023-03-24 01:56:32,366 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 9f2a3186a54c4b8eaaafc6b906bced90, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986271 2023-03-24 01:56:32,366 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6b7a15e9331548a889e85f01e82c7136, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679622986899 2023-03-24 01:56:32,370 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6618d6be9fe14ac2b59a5b1aac6269ae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679622987521 2023-03-24 01:56:32,370 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 671dafcd53a5498f8afb022d3f85db5e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679622986899 2023-03-24 01:56:32,370 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1ebdd4d0b31c43628bb4fb057c428f8e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679622987521 2023-03-24 01:56:32,370 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting c0ccb0189b534bdabf1bb757533be3f4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679622988675 2023-03-24 01:56:32,371 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b4245094f87746deb43048b07593dff4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679622988675 2023-03-24 01:56:32,371 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 330c359a86124ffb9c4b239486cc2dbf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679622989000 2023-03-24 01:56:32,371 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting b7d97d5bd2de443d81136c8fafe3ee42, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679622989000 2023-03-24 01:56:32,371 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8717fc4203fe46bc86c443140fb53f2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679622989630 2023-03-24 01:56:32,371 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1d1d66ac2b8f4d429cafad56c4cd40aa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679622989630 2023-03-24 01:56:32,372 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cac2cf3be3f9456aac3ae7893768e576, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:32,372 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0be12a6d5466401d968c995480be522f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:32,386 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#552 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:32,398 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#553 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:32,403 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6c5b5d0c8e0e48d2bb4506097426169d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6c5b5d0c8e0e48d2bb4506097426169d 2023-03-24 01:56:32,408 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 6c5b5d0c8e0e48d2bb4506097426169d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:32,409 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:32,409 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=9, startTime=1679622992362; duration=0sec 2023-03-24 01:56:32,409 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:32,409 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-24 01:56:32,411 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84108 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-24 01:56:32,411 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:32,411 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,411 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5d04ff6ac26d447f8eab4152de186e67, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=82.1 K 2023-03-24 01:56:32,411 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5d04ff6ac26d447f8eab4152de186e67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679622986271 2023-03-24 01:56:32,411 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 801f0bd4b018436a8c687c6cc277b7a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1679622986899 2023-03-24 01:56:32,412 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2d137cb35e3e4c08a2cee45a60443ef8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679622987521 2023-03-24 01:56:32,412 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1bf6d39f63964bdf8e2880cb06744c10, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679622988675 2023-03-24 01:56:32,412 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting f5af9df027f049dbbdb0fd82469b9577, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1679622989000 2023-03-24 01:56:32,413 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting b115050252c34cbf8796bb121a1b99da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1679622989630 2023-03-24 01:56:32,413 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 46fcd7c75b434ed1abc7e202e7c42a03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:32,422 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#554 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:32,433 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/7e94dbcd894943a7ae84de0e5e622311 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/7e94dbcd894943a7ae84de0e5e622311 2023-03-24 01:56:32,437 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 7e94dbcd894943a7ae84de0e5e622311(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:32,437 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:32,437 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=9, startTime=1679622992362; duration=0sec 2023-03-24 01:56:32,437 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:32,452 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:32,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:32,455 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:32,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:32,455 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:32,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,456 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,459 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,459 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,460 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:32,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:32,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:32,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:32,461 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:32,462 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:32,462 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:32,490 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324ffb3f184558e4444a180339ec0251e4c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ffb3f184558e4444a180339ec0251e4c 2023-03-24 01:56:32,495 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/33a4f721acf547a7b85ca93500149b27 2023-03-24 01:56:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:32,523 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:32,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49502 deadline: 1679623052534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623052537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623052538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623052639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623052640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49478 deadline: 1679623052642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,828 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/5fb272dd727d4b1e95a09460dc8c08e7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5fb272dd727d4b1e95a09460dc8c08e7 2023-03-24 01:56:32,833 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 5fb272dd727d4b1e95a09460dc8c08e7(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:32,833 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:32,833 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=9, startTime=1679622992362; duration=0sec 2023-03-24 01:56:32,833 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:32,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623052841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623052842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:32,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623052845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,905 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9b2236a1edf241969db80e51d8964ebd 2023-03-24 01:56:32,916 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/36178ea8adcf427e984ec569313574bb 2023-03-24 01:56:32,920 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/33a4f721acf547a7b85ca93500149b27 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27 2023-03-24 01:56:32,924 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27, entries=150, sequenceid=221, filesize=25.3 K 2023-03-24 01:56:32,925 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9b2236a1edf241969db80e51d8964ebd as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd 2023-03-24 01:56:32,928 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd, entries=150, sequenceid=221, filesize=11.8 K 2023-03-24 01:56:32,928 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/36178ea8adcf427e984ec569313574bb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb 2023-03-24 01:56:32,931 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb, entries=150, sequenceid=221, filesize=11.8 K 2023-03-24 01:56:32,932 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 470ms, sequenceid=221, compaction requested=false 2023-03-24 01:56:32,932 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:32,932 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:32,932 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:32,935 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,935 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4624ebfa[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:32,937 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:32,938 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,939 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:32,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:32,939 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:32,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,939 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:32,941 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:32,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,941 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:32,944 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:32,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:32,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:32,944 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:32,944 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:32,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,946 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:32,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:32,947 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,949 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:32,952 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:32,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:32,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:32,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:32,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:32,952 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,953 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:32,953 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:32,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:32,953 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,053 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:33,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:33,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:33,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:33,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:33,057 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:33,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:33,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:33,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:33,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,057 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,059 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:33,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:33,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,059 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:33,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:33,060 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:33,061 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:33,061 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:33,061 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:33,061 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:33,061 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:33,062 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,082 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241794f79055ac46a287d5eaee4f6ee368 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241794f79055ac46a287d5eaee4f6ee368 2023-03-24 01:56:33,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:33,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:33,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623053162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623053163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623053163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623053265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623053268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623053268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623053467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623053470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623053471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,488 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/0bda28c7e52b4c52a0a9298ed1c72ace 2023-03-24 01:56:33,497 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/99214f23904a4804836707259333ab8a 2023-03-24 01:56:33,508 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/2194008d94bb46e684e759e63f402043 2023-03-24 01:56:33,511 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/0bda28c7e52b4c52a0a9298ed1c72ace as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace 2023-03-24 01:56:33,514 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace, entries=150, sequenceid=237, filesize=25.3 K 2023-03-24 01:56:33,515 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/99214f23904a4804836707259333ab8a as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a 2023-03-24 01:56:33,518 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a, entries=150, sequenceid=237, filesize=11.8 K 2023-03-24 01:56:33,519 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/2194008d94bb46e684e759e63f402043 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043 2023-03-24 01:56:33,522 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043, entries=150, sequenceid=237, filesize=11.8 K 2023-03-24 01:56:33,523 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 462ms, sequenceid=237, compaction requested=true 2023-03-24 01:56:33,523 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:33,523 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:33,523 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:33,525 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,525 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:33,525 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,525 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:33,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cdbc79a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:33,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:33,527 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,528 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:33,528 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:33,528 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,528 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:33,530 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:33,530 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:33,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:33,534 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:33,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:33,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:33,534 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:33,534 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:33,535 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,535 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:33,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,535 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:33,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:33,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:33,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:33,536 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,538 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:33,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:33,540 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:33,541 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:33,541 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:33,540 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:33,541 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:33,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:33,641 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:33,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:33,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:33,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:33,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:33,644 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:33,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:33,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:33,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:33,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,645 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,646 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,646 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:33,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:33,647 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,647 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:33,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:33,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:33,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:33,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:33,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:33,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:33,648 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:33,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:33,649 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:33,649 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:33,649 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:33,666 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d4338b00d051490c80c74a0fcd3b3d91 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d4338b00d051490c80c74a0fcd3b3d91 2023-03-24 01:56:33,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:33,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:33,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623053786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623053787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623053788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623053889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623053889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:33,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:33,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623053891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,081 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/a3b5346dde8c4d5d87add25ce00cc10e 2023-03-24 01:56:34,093 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/137457ffa4754f8f8309bb70e8b6cdbe 2023-03-24 01:56:34,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623054093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623054094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,104 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/9d86f57d0d794ba2acb8d79dc8cc8a3f 2023-03-24 01:56:34,107 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/a3b5346dde8c4d5d87add25ce00cc10e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e 2023-03-24 01:56:34,112 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e, entries=150, sequenceid=254, filesize=25.3 K 2023-03-24 01:56:34,113 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/137457ffa4754f8f8309bb70e8b6cdbe as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe 2023-03-24 01:56:34,116 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 01:56:34,117 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/9d86f57d0d794ba2acb8d79dc8cc8a3f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f 2023-03-24 01:56:34,120 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 01:56:34,120 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 471ms, sequenceid=254, compaction requested=true 2023-03-24 01:56:34,120 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,120 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,120 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:34,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:34,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:34,121 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:34,122 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,122 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,123 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:34,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62285cc1[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:34,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:34,124 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,125 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:34,125 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,125 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:34,125 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:34,129 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:34,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:34,131 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:34,131 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:34,131 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:34,131 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:34,131 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:34,134 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,134 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,134 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,136 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,138 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,138 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:34,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:34,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,139 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,139 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:34,139 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:34,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,144 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,239 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:34,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:34,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:34,242 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:34,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:34,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:34,242 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:34,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,243 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,244 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,244 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:34,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:34,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:34,246 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:34,246 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:34,246 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,247 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:34,247 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:34,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:34,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623054417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623054418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623054522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623054522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,661 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324dcdf39f2c6f0430e84e51206fcf05aff to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dcdf39f2c6f0430e84e51206fcf05aff 2023-03-24 01:56:34,666 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1abbab42eeec48e1baf4c4a0fcce0dc8 2023-03-24 01:56:34,675 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/1a0297fff5d343bfb485cbb8a5951c3b 2023-03-24 01:56:34,693 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/cc64fb85d9db466d9363d774dc541d9c 2023-03-24 01:56:34,698 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1abbab42eeec48e1baf4c4a0fcce0dc8 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8 2023-03-24 01:56:34,701 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8, entries=150, sequenceid=269, filesize=25.4 K 2023-03-24 01:56:34,702 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/1a0297fff5d343bfb485cbb8a5951c3b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b 2023-03-24 01:56:34,709 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b, entries=150, sequenceid=269, filesize=11.9 K 2023-03-24 01:56:34,710 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/cc64fb85d9db466d9363d774dc541d9c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c 2023-03-24 01:56:34,713 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c, entries=150, sequenceid=269, filesize=11.9 K 2023-03-24 01:56:34,713 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 467ms, sequenceid=269, compaction requested=true 2023-03-24 01:56:34,713 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,713 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,713 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:34,715 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1b65b7[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:34,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:34,717 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,722 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:34,722 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:34,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:34,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 01:56:34,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:34,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,725 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:34,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:34,728 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:34,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:34,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:34,728 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:34,729 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:34,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,730 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,731 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,732 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,734 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:34,734 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:34,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,735 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,735 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:34,735 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623054743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623054746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,757 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032410370d2653554df8ba2289c269ea3ad1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032410370d2653554df8ba2289c269ea3ad1 2023-03-24 01:56:34,767 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/eaefe46583b9453197db86df4aad83d3 2023-03-24 01:56:34,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/3e85f814c60e4dfda547430715b70083 2023-03-24 01:56:34,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a4703d66cb004266a71896e7a1114cea 2023-03-24 01:56:34,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/eaefe46583b9453197db86df4aad83d3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3 2023-03-24 01:56:34,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3, entries=200, sequenceid=287, filesize=32.3 K 2023-03-24 01:56:34,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/3e85f814c60e4dfda547430715b70083 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083 2023-03-24 01:56:34,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083, entries=150, sequenceid=287, filesize=11.9 K 2023-03-24 01:56:34,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a4703d66cb004266a71896e7a1114cea as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea 2023-03-24 01:56:34,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea, entries=150, sequenceid=287, filesize=11.9 K 2023-03-24 01:56:34,818 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 94ms, sequenceid=287, compaction requested=true 2023-03-24 01:56:34,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:34,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:34,819 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:56:34,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:34,819 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:56:34,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163160 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:56:34,820 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:34,820 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,820 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5fb272dd727d4b1e95a09460dc8c08e7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=159.3 K 2023-03-24 01:56:34,820 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72960 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:56:34,820 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:34,820 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,821 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6c5b5d0c8e0e48d2bb4506097426169d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=71.3 K 2023-03-24 01:56:34,821 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5fb272dd727d4b1e95a09460dc8c08e7, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:34,821 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6c5b5d0c8e0e48d2bb4506097426169d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 33a4f721acf547a7b85ca93500149b27, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679622991896 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9b2236a1edf241969db80e51d8964ebd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679622991896 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0bda28c7e52b4c52a0a9298ed1c72ace, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679622992528 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 99214f23904a4804836707259333ab8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679622992528 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting a3b5346dde8c4d5d87add25ce00cc10e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679622993158 2023-03-24 01:56:34,822 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 137457ffa4754f8f8309bb70e8b6cdbe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679622993158 2023-03-24 01:56:34,823 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1abbab42eeec48e1baf4c4a0fcce0dc8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679622993785 2023-03-24 01:56:34,823 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 1a0297fff5d343bfb485cbb8a5951c3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679622993785 2023-03-24 01:56:34,823 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3e85f814c60e4dfda547430715b70083, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994413 2023-03-24 01:56:34,823 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting eaefe46583b9453197db86df4aad83d3, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994412 2023-03-24 01:56:34,835 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#570 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:34,835 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:34,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:34,839 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:34,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:34,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:34,839 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:34,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,840 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,841 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#571 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:34,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,843 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,843 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:34,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:34,847 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/86c839192d8c4b2fbe972b4ae01ee196 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/86c839192d8c4b2fbe972b4ae01ee196 2023-03-24 01:56:34,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:34,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 01:56:34,851 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 86c839192d8c4b2fbe972b4ae01ee196(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:34,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:34,851 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,852 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=10, startTime=1679622994818; duration=0sec 2023-03-24 01:56:34,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:34,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,852 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:34,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:34,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:34,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:34,852 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,852 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:56:34,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:34,853 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:34,854 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72960 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:56:34,854 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:34,854 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:34,854 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/7e94dbcd894943a7ae84de0e5e622311, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=71.3 K 2023-03-24 01:56:34,855 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 7e94dbcd894943a7ae84de0e5e622311, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1679622991770 2023-03-24 01:56:34,855 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,855 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 36178ea8adcf427e984ec569313574bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1679622991896 2023-03-24 01:56:34,855 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2194008d94bb46e684e759e63f402043, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1679622992528 2023-03-24 01:56:34,856 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9d86f57d0d794ba2acb8d79dc8cc8a3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679622993158 2023-03-24 01:56:34,856 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting cc64fb85d9db466d9363d774dc541d9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679622993785 2023-03-24 01:56:34,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,856 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:34,856 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,857 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a4703d66cb004266a71896e7a1114cea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994413 2023-03-24 01:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a812415[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:34,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:34,859 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,861 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:34,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:34,861 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:34,861 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,861 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:34,863 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,863 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,868 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324bcf602ba6dec4b37b57158110b96a09d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bcf602ba6dec4b37b57158110b96a09d 2023-03-24 01:56:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:34,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:34,869 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:34,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:34,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:34,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:34,869 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:34,871 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#573 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:34,874 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,874 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,874 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:34,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,875 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:34,876 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/31873ec1abf24a39be3f74a42818307c 2023-03-24 01:56:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,877 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:34,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623054874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:34,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:34,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:34,880 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:34,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:34,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:34,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:34,881 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:34,881 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:34,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:34,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:34,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623054879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,884 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/8b91b4aaac754922ac367342427c5c74 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8b91b4aaac754922ac367342427c5c74 2023-03-24 01:56:34,889 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 8b91b4aaac754922ac367342427c5c74(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:34,889 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,889 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=10, startTime=1679622994819; duration=0sec 2023-03-24 01:56:34,889 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:34,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6afbe461e9da4f26a7c351bd8ec97964 2023-03-24 01:56:34,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/8a9df47d7f62404597896c97d7ff4cac 2023-03-24 01:56:34,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/31873ec1abf24a39be3f74a42818307c as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c 2023-03-24 01:56:34,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c, entries=200, sequenceid=304, filesize=32.3 K 2023-03-24 01:56:34,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6afbe461e9da4f26a7c351bd8ec97964 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964 2023-03-24 01:56:34,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964, entries=150, sequenceid=304, filesize=11.9 K 2023-03-24 01:56:34,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/8a9df47d7f62404597896c97d7ff4cac as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac 2023-03-24 01:56:34,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac, entries=150, sequenceid=304, filesize=11.9 K 2023-03-24 01:56:34,916 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 65ms, sequenceid=304, compaction requested=false 2023-03-24 01:56:34,916 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:34,981 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:34,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:34,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:34,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:34,982 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:34,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:34,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:34,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:34,994 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,994 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,996 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:34,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:34,996 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:34,997 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:34,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:34,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:35,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:35,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:35,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:35,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,000 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:35,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623054996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,004 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:35,004 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:35,004 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,004 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:35,004 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:35,004 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,004 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:35,007 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,007 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,007 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,007 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:35,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a4c9c8[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:35,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:35,009 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,010 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324662276e9c81f42d181fc7430b3d45c1a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324662276e9c81f42d181fc7430b3d45c1a 2023-03-24 01:56:35,011 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:35,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,011 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:35,012 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,013 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:35,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:35,015 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:35,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:35,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:35,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:35,015 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:35,016 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,017 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,017 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,018 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,019 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,022 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:35,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:35,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:35,021 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,022 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:35,022 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:35,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623055102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,123 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:35,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:35,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:35,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:35,126 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:35,126 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,126 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:35,126 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:35,126 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,126 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,128 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,128 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,129 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:35,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:35,130 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,130 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:35,130 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:35,130 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:35,131 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:35,132 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8ca5649[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:35,134 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,135 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:35,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,135 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:35,141 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:35,143 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:35,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:35,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:35,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:35,143 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:35,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,145 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,145 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,147 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:35,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,150 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,150 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:35,150 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:35,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,250 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:35,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:35,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:35,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:35,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:35,255 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:35,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:35,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:35,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,255 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,256 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,257 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:35,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:35,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:35,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:35,258 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,258 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:35,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:35,259 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:35,261 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:35,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582cac3a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:35,264 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:35,264 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,265 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:35,265 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,265 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:35,267 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,267 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:35,270 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:35,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:35,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:35,270 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:35,270 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:35,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,272 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,273 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,274 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:35,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,277 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:35,277 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:35,277 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:35,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623055304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,377 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:35,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:35,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:35,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:35,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:35,380 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:35,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:35,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:35,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,381 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,383 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,383 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,383 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,383 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:35,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:35,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,384 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:35,385 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,385 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:35,385 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:35,386 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:35,387 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,388 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6e8aa6[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:35,389 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:35,390 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:35,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:35,394 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:35,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,394 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:35,397 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:35,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:35,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:35,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:35,397 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:35,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:35,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:35,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,403 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:35,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:35,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:35,404 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,404 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:35,404 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:35,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:35,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:35,404 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:35,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:35,416 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/bab2df183370437399c69718e0192639 2023-03-24 01:56:35,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/46ac818356864256ae2130e01dd93590 2023-03-24 01:56:35,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/05cc93e88756441ba164820227cb68ef 2023-03-24 01:56:35,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/bab2df183370437399c69718e0192639 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639 2023-03-24 01:56:35,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639, entries=150, sequenceid=322, filesize=25.4 K 2023-03-24 01:56:35,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/46ac818356864256ae2130e01dd93590 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590 2023-03-24 01:56:35,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590, entries=150, sequenceid=322, filesize=11.9 K 2023-03-24 01:56:35,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/05cc93e88756441ba164820227cb68ef as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef 2023-03-24 01:56:35,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef, entries=150, sequenceid=322, filesize=11.9 K 2023-03-24 01:56:35,467 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 485ms, sequenceid=322, compaction requested=true 2023-03-24 01:56:35,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:35,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:35,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:35,467 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 6 compacting, 2 eligible, 16 blocking 2023-03-24 01:56:35,467 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 01:56:35,468 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 01:56:35,468 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 01:56:35,468 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. because compaction request was cancelled 2023-03-24 01:56:35,468 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:35,469 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:35,469 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:35,469 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,469 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/86c839192d8c4b2fbe972b4ae01ee196, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=36.3 K 2023-03-24 01:56:35,469 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 86c839192d8c4b2fbe972b4ae01ee196, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994413 2023-03-24 01:56:35,470 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6afbe461e9da4f26a7c351bd8ec97964, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679622994740 2023-03-24 01:56:35,470 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 46ac818356864256ae2130e01dd93590, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622994868 2023-03-24 01:56:35,475 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:35,485 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/2517074e2ce34bd5babbab7e4cc94cb3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/2517074e2ce34bd5babbab7e4cc94cb3 2023-03-24 01:56:35,489 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 2517074e2ce34bd5babbab7e4cc94cb3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:35,489 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:35,489 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=13, startTime=1679622995467; duration=0sec 2023-03-24 01:56:35,489 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:35,489 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 01:56:35,490 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 01:56:35,490 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:35,490 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,490 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8b91b4aaac754922ac367342427c5c74, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=36.3 K 2023-03-24 01:56:35,490 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8b91b4aaac754922ac367342427c5c74, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994413 2023-03-24 01:56:35,491 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 8a9df47d7f62404597896c97d7ff4cac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679622994740 2023-03-24 01:56:35,491 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 05cc93e88756441ba164820227cb68ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622994868 2023-03-24 01:56:35,496 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:35,504 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:35,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:35,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:35,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:35,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:35,507 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:35,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:35,507 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:35,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,508 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,509 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:35,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:35,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:35,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,509 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:35,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:35,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:35,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:35,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:35,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:35,511 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:35,512 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:35,513 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:35,532 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d452566ca94a440eadd739cfb20a8774 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d452566ca94a440eadd739cfb20a8774 2023-03-24 01:56:35,537 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/3beefc407096403eaebda0806c698c83 2023-03-24 01:56:35,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:35,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:35,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623055630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,665 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/4b6addbb9e3c489aa1fbc3c4b7ebe292 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4b6addbb9e3c489aa1fbc3c4b7ebe292 2023-03-24 01:56:35,670 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 4b6addbb9e3c489aa1fbc3c4b7ebe292(size=26.1 K), total size for store is 83.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:35,670 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:35,670 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=10, startTime=1679622994818; duration=0sec 2023-03-24 01:56:35,670 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:35,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623055734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,905 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/414750f193bb46c2b3f5f7930f281c21 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/414750f193bb46c2b3f5f7930f281c21 2023-03-24 01:56:35,910 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 414750f193bb46c2b3f5f7930f281c21(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:35,910 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:35,910 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=13, startTime=1679622995467; duration=0sec 2023-03-24 01:56:35,910 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:35,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623055938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623055938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:35,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623055941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:35,950 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/5dfd36efff6a4ef99dfd19223e2b49c5 2023-03-24 01:56:36,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623056240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623056241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623056245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,361 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/665c83a0ab6340059ec1b89d645f1eed 2023-03-24 01:56:36,365 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/3beefc407096403eaebda0806c698c83 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83 2023-03-24 01:56:36,369 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83, entries=150, sequenceid=338, filesize=25.4 K 2023-03-24 01:56:36,370 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/5dfd36efff6a4ef99dfd19223e2b49c5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5 2023-03-24 01:56:36,373 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5, entries=150, sequenceid=338, filesize=11.9 K 2023-03-24 01:56:36,374 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/665c83a0ab6340059ec1b89d645f1eed as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed 2023-03-24 01:56:36,378 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed, entries=150, sequenceid=338, filesize=11.9 K 2023-03-24 01:56:36,379 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 867ms, sequenceid=338, compaction requested=true 2023-03-24 01:56:36,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:36,379 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:36,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:36,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:36,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:36,379 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,382 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,382 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:36,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:36,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:36,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@358afe8a[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:36,383 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:36,384 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,385 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:36,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:36,385 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:36,385 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,385 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:36,386 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:36,387 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:36,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:36,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:36,389 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:36,389 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:36,389 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:36,389 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:36,389 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:36,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:36,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,391 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:36,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:36,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,392 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,393 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:36,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:36,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:36,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:36,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:36,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:36,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:36,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:36,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:36,398 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:36,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:36,399 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,399 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:36,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:36,399 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:36,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:36,499 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:36,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:36,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:36,502 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:36,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:36,502 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:36,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:36,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,503 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:36,504 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:36,504 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:36,505 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,505 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:36,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:36,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:36,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:36,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:36,506 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:36,506 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:36,506 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:36,506 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:36,507 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:36,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:36,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:36,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623056757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623056760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623056761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623056861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623056864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:36,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623056864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:36,921 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249c546bc760cc4585a7e740e0db5992a6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c546bc760cc4585a7e740e0db5992a6 2023-03-24 01:56:37,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623057065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623057068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623057069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,326 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2 2023-03-24 01:56:37,337 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9bc30a25b0af4bedb46b3030ba039260 2023-03-24 01:56:37,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623057368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623057372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623057373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,747 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/e74acc500b88430e9c604f88ba824adc 2023-03-24 01:56:37,750 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2 2023-03-24 01:56:37,753 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2, entries=150, sequenceid=357, filesize=25.4 K 2023-03-24 01:56:37,754 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9bc30a25b0af4bedb46b3030ba039260 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260 2023-03-24 01:56:37,757 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260, entries=150, sequenceid=357, filesize=11.9 K 2023-03-24 01:56:37,758 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/e74acc500b88430e9c604f88ba824adc as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc 2023-03-24 01:56:37,761 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc, entries=150, sequenceid=357, filesize=11.9 K 2023-03-24 01:56:37,762 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 1256ms, sequenceid=357, compaction requested=true 2023-03-24 01:56:37,762 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:37,762 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:37,764 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:37,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f288e6c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:37,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:37,766 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,767 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,767 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:37,767 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:37,767 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:37,768 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,768 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:37,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:37,770 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,770 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:37,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:37,773 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:37,773 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:37,774 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:37,774 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:37,774 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:37,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,775 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:37,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:37,776 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,777 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,780 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:37,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:37,780 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:37,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:37,781 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:37,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:37,781 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:37,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:37,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:37,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:37,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:37,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:37,881 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:37,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:37,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:37,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:37,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:37,885 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:37,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:37,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:37,885 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:37,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,886 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,887 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:37,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:37,887 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:37,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,888 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:37,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:37,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:37,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:37,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:37,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:37,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:37,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:37,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:37,889 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:37,889 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:37,889 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:37,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:37,890 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:37,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:37,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:37,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:37,890 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:37,890 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032425aed75d60ca4e88bd43a11d84948cc5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032425aed75d60ca4e88bd43a11d84948cc5 2023-03-24 01:56:37,892 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,892 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,893 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:37,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5258b2cf[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:37,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:37,895 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,895 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/3cd8f632976848bf84941aa1011f1acb 2023-03-24 01:56:37,897 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:37,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:37,897 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:37,898 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,898 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:37,899 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:37,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:37,900 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623057897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:37,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:37,903 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:37,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:37,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:37,903 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:37,903 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:37,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623057898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:37,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623057899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,905 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:37,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:37,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,906 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:37,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:37,908 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:37,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/356cf0377ad44cd9b947c3cf2ac3e048 2023-03-24 01:56:37,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:37,911 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:37,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:37,911 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:37,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:37,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:37,911 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:37,912 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:38,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623058002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623058004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623058004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,012 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:38,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:38,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:38,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:38,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:38,015 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:38,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:38,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:38,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,015 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,017 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,017 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:38,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:38,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:38,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:38,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:38,018 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:38,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:38,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:38,019 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:38,019 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:38,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,021 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,021 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:38,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dac3998[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:38,023 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:38,023 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,024 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:38,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:38,024 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:38,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,024 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:38,027 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:38,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,027 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:38,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:38,029 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:38,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:38,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:38,029 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:38,029 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:38,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,031 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,032 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,033 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,039 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:38,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:38,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,040 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,039 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,040 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:38,040 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:38,040 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,140 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:38,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:38,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:38,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:38,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:38,143 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:38,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:38,143 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:38,144 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,144 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,146 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,146 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,146 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,146 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:38,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:38,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:38,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:38,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:38,147 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:38,147 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,147 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:38,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:38,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:38,148 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:38,148 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:38,149 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,150 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,150 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,150 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:38,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f30fa09[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:38,152 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:38,152 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,153 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,153 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:38,153 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:38,153 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:38,153 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,153 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:38,154 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:38,154 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,155 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:38,155 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:38,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:38,157 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:38,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:38,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:38,157 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:38,158 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:38,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,160 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,161 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,162 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,164 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,165 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,165 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:38,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:38,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,168 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:38,168 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:38,168 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:38,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623058206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623058207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623058207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,268 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:38,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:38,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:38,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:38,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:38,272 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:38,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:38,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:38,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,272 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,273 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,274 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,274 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,274 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:38,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:38,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:38,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:38,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:38,275 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:38,275 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,275 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:38,276 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:38,276 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:38,276 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:38,277 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,277 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,278 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:38,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5440406d[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:38,280 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:38,280 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,281 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:38,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,282 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:38,282 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:38,284 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,284 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:38,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:38,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:38,287 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:38,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:38,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:38,287 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:38,287 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:38,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,288 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,289 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,290 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,293 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:38,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:38,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,293 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,293 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:38,293 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:38,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/86407e84243345d8a5d6ee0e219b550d 2023-03-24 01:56:38,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/3cd8f632976848bf84941aa1011f1acb as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb 2023-03-24 01:56:38,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb, entries=200, sequenceid=373, filesize=32.3 K 2023-03-24 01:56:38,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/356cf0377ad44cd9b947c3cf2ac3e048 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048 2023-03-24 01:56:38,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048, entries=150, sequenceid=373, filesize=11.9 K 2023-03-24 01:56:38,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/86407e84243345d8a5d6ee0e219b550d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d 2023-03-24 01:56:38,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d, entries=150, sequenceid=373, filesize=11.9 K 2023-03-24 01:56:38,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 459ms, sequenceid=373, compaction requested=true 2023-03-24 01:56:38,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:38,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:38,337 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 01:56:38,337 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:38,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:38,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:38,338 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:38,338 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:38,338 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,338 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/2517074e2ce34bd5babbab7e4cc94cb3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=48.3 K 2023-03-24 01:56:38,339 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 2517074e2ce34bd5babbab7e4cc94cb3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622994868 2023-03-24 01:56:38,339 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 171004 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 01:56:38,339 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:38,339 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,339 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4b6addbb9e3c489aa1fbc3c4b7ebe292, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=167.0 K 2023-03-24 01:56:38,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5dfd36efff6a4ef99dfd19223e2b49c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679622994992 2023-03-24 01:56:38,340 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4b6addbb9e3c489aa1fbc3c4b7ebe292, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1679622994413 2023-03-24 01:56:38,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9bc30a25b0af4bedb46b3030ba039260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622995621 2023-03-24 01:56:38,340 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 31873ec1abf24a39be3f74a42818307c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679622994740 2023-03-24 01:56:38,340 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 356cf0377ad44cd9b947c3cf2ac3e048, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996759 2023-03-24 01:56:38,341 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting bab2df183370437399c69718e0192639, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622994868 2023-03-24 01:56:38,341 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 3beefc407096403eaebda0806c698c83, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679622994992 2023-03-24 01:56:38,341 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting d1c3a7cb2b1c451397be4cb0ccb2f3a2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622995621 2023-03-24 01:56:38,342 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 3cd8f632976848bf84941aa1011f1acb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996754 2023-03-24 01:56:38,347 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#590 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:38,362 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#591 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:38,366 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/e43f92ef2fe745af9237bd91c2bb10ff as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/e43f92ef2fe745af9237bd91c2bb10ff 2023-03-24 01:56:38,370 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into e43f92ef2fe745af9237bd91c2bb10ff(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:38,370 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:38,370 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=12, startTime=1679622998337; duration=0sec 2023-03-24 01:56:38,370 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:38,370 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:38,371 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:38,371 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:38,371 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,371 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/414750f193bb46c2b3f5f7930f281c21, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=48.3 K 2023-03-24 01:56:38,372 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 414750f193bb46c2b3f5f7930f281c21, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1679622994868 2023-03-24 01:56:38,372 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 665c83a0ab6340059ec1b89d645f1eed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1679622994992 2023-03-24 01:56:38,373 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e74acc500b88430e9c604f88ba824adc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1679622995621 2023-03-24 01:56:38,373 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 86407e84243345d8a5d6ee0e219b550d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996759 2023-03-24 01:56:38,390 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#592 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:38,393 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:38,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:38,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:38,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:38,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:38,397 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:38,397 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:38,398 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:38,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,399 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,401 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,401 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,402 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,402 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:38,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:38,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:38,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:38,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:38,404 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:38,404 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,404 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/85c6cf5c92c54013af069b45535c0ba1 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/85c6cf5c92c54013af069b45535c0ba1 2023-03-24 01:56:38,404 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:38,407 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:38,407 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:38,407 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,407 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:38,407 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:38,408 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 85c6cf5c92c54013af069b45535c0ba1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:38,408 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:38,408 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=12, startTime=1679622998337; duration=0sec 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:38,408 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:38,408 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:38,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:38,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:38,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623058524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623058526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623058527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623058629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623058630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623058630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,800 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/0d7214bef8ee4f3b80d5c7958fcf4ff7 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0d7214bef8ee4f3b80d5c7958fcf4ff7 2023-03-24 01:56:38,804 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 0d7214bef8ee4f3b80d5c7958fcf4ff7(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:38,804 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:38,804 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=10, startTime=1679622998337; duration=0sec 2023-03-24 01:56:38,804 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:38,823 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032476f0f590f0fc49f3ac5d1d1f5268eae9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032476f0f590f0fc49f3ac5d1d1f5268eae9 2023-03-24 01:56:38,827 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1f614285bde244fdbb46348bb2ad9865 2023-03-24 01:56:38,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623058832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623058833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:38,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623058834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,836 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/ad6259e61cba4202a09cddc3d4ed172e 2023-03-24 01:56:38,845 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/70b9fb8ff9b341b2925a4736e86ac9db 2023-03-24 01:56:38,849 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/1f614285bde244fdbb46348bb2ad9865 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865 2023-03-24 01:56:38,852 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865, entries=150, sequenceid=391, filesize=25.4 K 2023-03-24 01:56:38,853 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/ad6259e61cba4202a09cddc3d4ed172e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e 2023-03-24 01:56:38,857 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e, entries=150, sequenceid=391, filesize=11.9 K 2023-03-24 01:56:38,860 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/70b9fb8ff9b341b2925a4736e86ac9db as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db 2023-03-24 01:56:38,863 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db, entries=150, sequenceid=391, filesize=11.9 K 2023-03-24 01:56:38,864 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 457ms, sequenceid=391, compaction requested=false 2023-03-24 01:56:38,864 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:38,864 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,867 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:38,867 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43194ee0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:38,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:38,869 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,870 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:38,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,871 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:38,871 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:38,874 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:38,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:38,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:38,874 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:38,874 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:38,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,877 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:38,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,877 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,879 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:38,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:38,881 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:38,881 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:38,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:38,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,881 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:38,881 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:38,882 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:38,981 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:38,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:38,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:38,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:38,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:38,985 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:38,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:38,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:38,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,985 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,989 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:38,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:38,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:38,989 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:38,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:38,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:38,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:38,991 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:38,991 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:38,991 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:38,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:38,992 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:39,009 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244847db0673b24fedb122d4c718697044 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244847db0673b24fedb122d4c718697044 2023-03-24 01:56:39,017 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/91ca3ec6275042fa9c8b6665e686a15d 2023-03-24 01:56:39,026 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6759ed596c3743ca8c8dd05929bbcf00 2023-03-24 01:56:39,037 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/79f08acee97a4b4ab25e34d0ac2ad1db 2023-03-24 01:56:39,041 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/91ca3ec6275042fa9c8b6665e686a15d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d 2023-03-24 01:56:39,045 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d, entries=150, sequenceid=408, filesize=25.4 K 2023-03-24 01:56:39,045 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/6759ed596c3743ca8c8dd05929bbcf00 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00 2023-03-24 01:56:39,049 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00, entries=150, sequenceid=408, filesize=11.9 K 2023-03-24 01:56:39,049 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/79f08acee97a4b4ab25e34d0ac2ad1db as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db 2023-03-24 01:56:39,053 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db, entries=150, sequenceid=408, filesize=11.9 K 2023-03-24 01:56:39,053 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 62ms, sequenceid=408, compaction requested=true 2023-03-24 01:56:39,053 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:39,053 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:39,053 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,055 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,055 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f85cfe2[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:39,073 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:39,074 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,076 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:39,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:39,076 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:39,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,076 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:39,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:39,077 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,077 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,078 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:39,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:39,091 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:39,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:39,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:39,092 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:39,092 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:39,093 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,095 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,096 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:39,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:39,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,099 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,100 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,100 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,100 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:39,100 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:39,099 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,100 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:39,156 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:56:39,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:39,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:39,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:39,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:39,157 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:39,157 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:39,178 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324322936ff55594881978bc84ad59f3d82 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324322936ff55594881978bc84ad59f3d82 2023-03-24 01:56:39,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679622999165, max timestamp=1679622999184Num uniques 859; 2023-03-24 01:56:39,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 01:56:39,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679622999165, max timestamp=1679622999184Num uniques 859; 2023-03-24 01:56:39,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:39,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679622999165, max timestamp=1679622999184Num uniques 859; 2023-03-24 01:56:39,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 01:56:39,200 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:39,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:39,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:39,203 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:39,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:39,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:39,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:39,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:39,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:39,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:39,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:39,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:39,207 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:39,207 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,207 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:39,208 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:39,208 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:39,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:39,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:39,208 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:39,209 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,209 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,210 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:39,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@479d9aa9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:39,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:39,212 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,213 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:39,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:39,213 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:39,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,214 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:39,215 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:39,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:39,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:39,218 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:39,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:39,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:39,218 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:39,219 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:39,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623059216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,221 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,222 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623059218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623059219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,223 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,226 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:39,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:39,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,226 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,227 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:39,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,227 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623059321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623059323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,327 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:39,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623059323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:39,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:39,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:39,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:39,338 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:39,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:39,338 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:39,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,339 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,341 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,341 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:39,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:39,345 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,345 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:39,345 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:39,345 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:39,345 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:39,347 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,347 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:39,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1188e91f[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:39,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:39,349 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,351 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:39,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:39,351 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:39,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,351 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:39,352 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,352 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:39,352 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,352 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:39,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:39,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:39,354 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:39,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:39,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:39,354 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:39,354 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:39,356 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,356 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,356 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,357 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,358 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,360 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,359 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:39,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:39,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,360 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,360 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:39,360 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:39,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:39,460 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:39,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:39,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:39,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:39,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:39,463 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:39,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:39,463 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:39,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,464 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,466 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,466 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,467 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:39,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:39,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:39,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:39,467 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:39,468 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,468 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:39,468 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:39,468 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:39,468 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:39,469 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,469 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,470 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:39,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18efdb80[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:39,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:39,472 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,476 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:39,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,477 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:39,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:39,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,478 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,478 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:39,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:39,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:39,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:39,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:39,481 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:39,481 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:39,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,482 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,483 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,483 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,484 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,487 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,488 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:39,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:39,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,488 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,489 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:39,489 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:39,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:39,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623059525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623059528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623059529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,589 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:39,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:39,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:39,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:39,592 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:39,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:39,592 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:39,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,593 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,593 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=418, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/6d01e24ede804de487a19960ec5d3e54 2023-03-24 01:56:39,594 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,594 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,595 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:39,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:39,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:39,595 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:39,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:39,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:39,596 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:39,596 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:39,597 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 01:56:39,598 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,598 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:39,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@904eda0[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:39,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:39,601 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,603 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:39,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:39,603 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:39,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,603 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:39,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/835d56caad354f7883c8171fc969ab5e 2023-03-24 01:56:39,606 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:39,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,606 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:39,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:39,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:39,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:39,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:39,609 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:39,609 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:39,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,611 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:39,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:39,612 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:39,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,614 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a7160495270843bc86eb4c378f86cbf4 2023-03-24 01:56:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:39,617 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:39,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:39,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:39,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/6d01e24ede804de487a19960ec5d3e54 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54 2023-03-24 01:56:39,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:39,617 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,618 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:39,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:39,618 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:39,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54, entries=350, sequenceid=418, filesize=52.9 K 2023-03-24 01:56:39,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/835d56caad354f7883c8171fc969ab5e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e 2023-03-24 01:56:39,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e, entries=150, sequenceid=418, filesize=11.9 K 2023-03-24 01:56:39,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a7160495270843bc86eb4c378f86cbf4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4 2023-03-24 01:56:39,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4, entries=150, sequenceid=418, filesize=11.9 K 2023-03-24 01:56:39,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.55 KB/134705 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 473ms, sequenceid=418, compaction requested=true 2023-03-24 01:56:39,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:39,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:39,629 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:39,629 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:39,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:39,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:39,630 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 133204 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:39,630 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:39,630 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:39,630 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,630 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:39,630 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0d7214bef8ee4f3b80d5c7958fcf4ff7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=130.1 K 2023-03-24 01:56:39,630 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,630 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/e43f92ef2fe745af9237bd91c2bb10ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=48.5 K 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 0d7214bef8ee4f3b80d5c7958fcf4ff7, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996759 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting e43f92ef2fe745af9237bd91c2bb10ff, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996759 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 1f614285bde244fdbb46348bb2ad9865, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679622997888 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting ad6259e61cba4202a09cddc3d4ed172e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679622997888 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 91ca3ec6275042fa9c8b6665e686a15d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622998522 2023-03-24 01:56:39,631 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 6759ed596c3743ca8c8dd05929bbcf00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622998522 2023-03-24 01:56:39,632 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6d01e24ede804de487a19960ec5d3e54, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:39,632 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 835d56caad354f7883c8171fc969ab5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:39,645 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#603 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:39,647 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#602 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:39,718 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:39,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:39,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:39,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:39,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:39,722 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:39,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:39,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:39,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,722 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,724 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:39,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,724 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:39,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:39,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:39,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:39,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:39,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:39,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:39,725 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:39,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:39,726 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:39,726 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:39,726 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:39,739 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032417ace63ed67e4e6fa216b642f8e0f428 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032417ace63ed67e4e6fa216b642f8e0f428 2023-03-24 01:56:39,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:39,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:39,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623059840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623059840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623059843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623059944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623059944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:39,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623059948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,066 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/4b0d50c0c6df4f22861a183ea4a7b5cf as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/4b0d50c0c6df4f22861a183ea4a7b5cf 2023-03-24 01:56:40,070 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 4b0d50c0c6df4f22861a183ea4a7b5cf(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:40,070 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:40,070 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=12, startTime=1679622999629; duration=0sec 2023-03-24 01:56:40,071 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:40,071 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 01:56:40,072 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 01:56:40,072 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:40,072 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:40,072 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/85c6cf5c92c54013af069b45535c0ba1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=48.5 K 2023-03-24 01:56:40,073 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 85c6cf5c92c54013af069b45535c0ba1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679622996759 2023-03-24 01:56:40,073 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 70b9fb8ff9b341b2925a4736e86ac9db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679622997888 2023-03-24 01:56:40,073 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 79f08acee97a4b4ab25e34d0ac2ad1db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679622998522 2023-03-24 01:56:40,073 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a7160495270843bc86eb4c378f86cbf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:40,080 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#605 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:40,144 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/5ba29a3b99344f2494474fd3796f203d 2023-03-24 01:56:40,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623060146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623060146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623060150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,154 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/5154507cb7774b5eb4fe59126bda68fa 2023-03-24 01:56:40,169 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/91a4ebca2b1e4f479791eb231e77220b 2023-03-24 01:56:40,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/5ba29a3b99344f2494474fd3796f203d as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d 2023-03-24 01:56:40,176 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d, entries=150, sequenceid=454, filesize=25.4 K 2023-03-24 01:56:40,177 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/5154507cb7774b5eb4fe59126bda68fa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa 2023-03-24 01:56:40,180 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa, entries=150, sequenceid=454, filesize=11.9 K 2023-03-24 01:56:40,181 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/91a4ebca2b1e4f479791eb231e77220b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b 2023-03-24 01:56:40,185 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b, entries=150, sequenceid=454, filesize=11.9 K 2023-03-24 01:56:40,186 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 460ms, sequenceid=454, compaction requested=false 2023-03-24 01:56:40,186 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:40,186 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:40,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:40,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:40,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:40,186 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:40,188 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:40,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:40,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:40,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:40,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226766fb[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:40,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:40,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,194 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:40,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,195 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:40,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:40,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:40,196 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,196 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:40,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:40,198 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:40,198 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:40,198 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:40,198 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:40,198 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:40,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:40,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,200 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:40,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:40,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,201 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:40,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:40,206 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:40,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:40,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:40,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:40,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:40,206 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:40,206 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:40,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:40,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:40,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:40,207 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:40,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:40,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:40,307 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:40,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:40,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:40,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:40,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:40,312 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:40,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:40,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:40,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:40,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,312 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:40,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:40,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:40,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:40,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:40,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,314 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:40,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:40,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:40,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:40,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:40,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:40,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:40,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:40,316 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 01:56:40,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:40,317 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:40,334 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f7da68ff4a204cef92f6deedc9cae1c7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7da68ff4a204cef92f6deedc9cae1c7 2023-03-24 01:56:40,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:40,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:40,469 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/6db59438202046c5a3590ad90a8df312 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6db59438202046c5a3590ad90a8df312 2023-03-24 01:56:40,473 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 6db59438202046c5a3590ad90a8df312(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:40,473 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:40,473 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=12, startTime=1679622999629; duration=0sec 2023-03-24 01:56:40,473 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:40,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623000471, max timestamp=1679623000478Num uniques 859; 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623000471, max timestamp=1679623000478Num uniques 859; 2023-03-24 01:56:40,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623060478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623000471, max timestamp=1679623000478Num uniques 859; 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:40,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:40,488 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/020654c0a87549d3901523177615bc88 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/020654c0a87549d3901523177615bc88 2023-03-24 01:56:40,492 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 020654c0a87549d3901523177615bc88(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:40,492 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:40,492 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=12, startTime=1679622999629; duration=0sec 2023-03-24 01:56:40,492 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:40,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623060507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623060508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49480 deadline: 1679623060542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,546 DEBUG [Thread-3894] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:40,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623060583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:49508 deadline: 1679623060584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,588 DEBUG [Thread-3892] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18218 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., hostname=jenkins-hbase4.apache.org,44439,1679622817494, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 01:56:40,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623060610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623060611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,745 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/e12f524d5e0a4be1a505f56bc6f7d0fa 2023-03-24 01:56:40,754 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9ccb22ba3e0f4025806e8eda99177210 2023-03-24 01:56:40,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623060784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623060814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:40,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623060815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623061089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623061118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623061120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,172 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/3237788dde304070a06a8039866a5cd5 2023-03-24 01:56:41,175 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/e12f524d5e0a4be1a505f56bc6f7d0fa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa 2023-03-24 01:56:41,178 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa, entries=150, sequenceid=465, filesize=25.4 K 2023-03-24 01:56:41,179 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/9ccb22ba3e0f4025806e8eda99177210 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210 2023-03-24 01:56:41,182 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210, entries=150, sequenceid=465, filesize=11.9 K 2023-03-24 01:56:41,183 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/3237788dde304070a06a8039866a5cd5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5 2023-03-24 01:56:41,186 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5, entries=150, sequenceid=465, filesize=11.9 K 2023-03-24 01:56:41,187 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 871ms, sequenceid=465, compaction requested=true 2023-03-24 01:56:41,187 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:41,187 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:41,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:41,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:41,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:41,187 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:41,189 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:41,189 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:41,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,190 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:41,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:41,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7471a3b9[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:41,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:41,191 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,193 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:41,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,193 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:41,197 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:41,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:41,200 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:41,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:41,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:41,200 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:41,200 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:41,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,201 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:41,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,202 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:41,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,203 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:41,205 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:41,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:41,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:41,205 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:41,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:41,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:41,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:41,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:41,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:41,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:41,206 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,206 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:41,206 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:41,306 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:41,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:41,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-24 01:56:41,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 01:56:41,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 01:56:41,310 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 01:56:41,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:41,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 01:56:41,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 01:56:41,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,310 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:41,313 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:41,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:41,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:41,313 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 01:56:41,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 01:56:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:41,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 01:56:41,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 01:56:41,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 01:56:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 01:56:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 01:56:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 01:56:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 01:56:41,315 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 01:56:41,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:41,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. started... 2023-03-24 01:56:41,315 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 01:56:41,315 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:41,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:41,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:41,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:41,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:41,316 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:41,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:41,595 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. as already flushing 2023-03-24 01:56:41,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623061614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49510 deadline: 1679623061623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49478 deadline: 1679623061627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=44439] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623061718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:41,735 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032461f18810f2aa496ab897891fa09202f9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461f18810f2aa496ab897891fa09202f9 2023-03-24 01:56:41,739 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=504, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/8f66a3802aca492aabb928eddf72df4f 2023-03-24 01:56:41,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 01:56:41,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=44439] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:49502 deadline: 1679623061922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,148 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/766a32bfe14044dc96fc114977ce1e58 2023-03-24 01:56:42,158 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/5bc0fa3b575140d4b94a40c2bfa12062 2023-03-24 01:56:42,161 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/8f66a3802aca492aabb928eddf72df4f as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f 2023-03-24 01:56:42,164 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f, entries=150, sequenceid=504, filesize=25.4 K 2023-03-24 01:56:42,165 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/766a32bfe14044dc96fc114977ce1e58 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58 2023-03-24 01:56:42,169 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58, entries=150, sequenceid=504, filesize=11.9 K 2023-03-24 01:56:42,169 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/5bc0fa3b575140d4b94a40c2bfa12062 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062 2023-03-24 01:56:42,173 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062, entries=150, sequenceid=504, filesize=11.9 K 2023-03-24 01:56:42,173 INFO [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 858ms, sequenceid=504, compaction requested=true 2023-03-24 01:56:42,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:42,173 DEBUG [rs(jenkins-hbase4.apache.org,44439,1679622817494)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:42,173 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 01:56:42,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 01:56:42,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 01:56:42,174 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 01:56:42,175 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,176 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 01:56:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44439,1679622817494' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 01:56:42,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a4456c[Count = 0] remaining members to acquire global barrier 2023-03-24 01:56:42,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 01:56:42,177 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,178 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 01:56:42,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 01:56:42,178 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,44439,1679622817494' in zk 2023-03-24 01:56:42,178 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,178 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 01:56:42,180 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 01:56:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,44439,1679622817494' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 01:56:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,44439,1679622817494': 2023-03-24 01:56:42,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44439,1679622817494' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 01:56:42,183 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 01:56:42,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 01:56:42,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 01:56:42,183 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 01:56:42,183 INFO [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 01:56:42,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 01:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 01:56:42,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,186 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:42,186 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 01:56:42,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 01:56:42,188 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 01:56:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 01:56:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 01:56:42,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 01:56:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 01:56:42,190 DEBUG [(jenkins-hbase4.apache.org,35345,1679622816283)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 01:56:42,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 01:56:42,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 01:56:42,190 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:42,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 01:56:42,191 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 01:56:42,191 DEBUG [Thread-3900] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 01:56:42,191 DEBUG [Thread-3900] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 01:56:42,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 01:56:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 01:56:42,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 01:56:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44439] regionserver.HRegion(9158): Flush requested on 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:42,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:42,228 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 01:56:42,290 DEBUG [Thread-3901] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:62686 2023-03-24 01:56:42,290 DEBUG [Thread-3901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,291 DEBUG [Thread-3900] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 01:56:42,291 DEBUG [Thread-3898] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:62686 2023-03-24 01:56:42,291 DEBUG [Thread-3898] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 01:56:42,291 DEBUG [Thread-3905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:62686 2023-03-24 01:56:42,291 DEBUG [Thread-3903] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:62686 2023-03-24 01:56:42,292 DEBUG [Thread-3905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,292 DEBUG [Thread-3903] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,295 DEBUG [Thread-3909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:62686 2023-03-24 01:56:42,295 DEBUG [Thread-3909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,297 DEBUG [Thread-3907] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:62686 2023-03-24 01:56:42,298 DEBUG [Thread-3907] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,632 DEBUG [Thread-3896] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:62686 2023-03-24 01:56:42,633 DEBUG [Thread-3896] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,639 DEBUG [Thread-3890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:62686 2023-03-24 01:56:42,639 DEBUG [Thread-3890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:42,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623002277, max timestamp=1679623002638Num uniques 859; 2023-03-24 01:56:42,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623002277, max timestamp=1679623002638Num uniques 859; 2023-03-24 01:56:42,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:42,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679623002277, max timestamp=1679623002638Num uniques 859; 2023-03-24 01:56:42,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:42,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 01:56:42,660 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d8d55eaf09f44b3588951215e065dca4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d8d55eaf09f44b3588951215e065dca4 2023-03-24 01:56:43,065 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=515, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/4649a754206740a6a69c24dade20e0b5 2023-03-24 01:56:43,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/04a2f2ef40734f5c81ec3a710f9f404e 2023-03-24 01:56:43,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a6a5bf5eebed47148c4b3d81ab2e7823 2023-03-24 01:56:43,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/4649a754206740a6a69c24dade20e0b5 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5 2023-03-24 01:56:43,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5, entries=150, sequenceid=515, filesize=25.4 K 2023-03-24 01:56:43,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/04a2f2ef40734f5c81ec3a710f9f404e as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e 2023-03-24 01:56:43,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e, entries=100, sequenceid=515, filesize=9.5 K 2023-03-24 01:56:43,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/a6a5bf5eebed47148c4b3d81ab2e7823 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823 2023-03-24 01:56:43,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823, entries=100, sequenceid=515, filesize=9.5 K 2023-03-24 01:56:43,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 867ms, sequenceid=515, compaction requested=true 2023-03-24 01:56:43,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:43,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:43,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:43,094 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:56:43,094 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:56:43,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:43,095 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 131214 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:56:43,095 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/A is initiating minor compaction (all files) 2023-03-24 01:56:43,095 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59508 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:56:43,095 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/A in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:43,095 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/B is initiating minor compaction (all files) 2023-03-24 01:56:43,095 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6db59438202046c5a3590ad90a8df312, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=128.1 K 2023-03-24 01:56:43,095 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/B in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:43,095 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/4b0d50c0c6df4f22861a183ea4a7b5cf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=58.1 K 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 6db59438202046c5a3590ad90a8df312, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 4b0d50c0c6df4f22861a183ea4a7b5cf, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 5ba29a3b99344f2494474fd3796f203d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679622999213 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5154507cb7774b5eb4fe59126bda68fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679622999213 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting e12f524d5e0a4be1a505f56bc6f7d0fa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679622999835 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 9ccb22ba3e0f4025806e8eda99177210, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679622999835 2023-03-24 01:56:43,096 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 8f66a3802aca492aabb928eddf72df4f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679623000499 2023-03-24 01:56:43,097 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 766a32bfe14044dc96fc114977ce1e58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679623000499 2023-03-24 01:56:43,097 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] compactions.Compactor(207): Compacting 4649a754206740a6a69c24dade20e0b5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679623001610 2023-03-24 01:56:43,097 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 04a2f2ef40734f5c81ec3a710f9f404e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679623001610 2023-03-24 01:56:43,105 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#B#compaction#618 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:43,106 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#A#compaction#617 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:43,512 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/236bdf41542e4606a4da0c511ffb088b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/236bdf41542e4606a4da0c511ffb088b 2023-03-24 01:56:43,516 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/B of 4c6892182a9cd7f5a6bb4c2432955ea0 into 236bdf41542e4606a4da0c511ffb088b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:43,516 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:43,516 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/B, priority=11, startTime=1679623003094; duration=0sec 2023-03-24 01:56:43,516 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 01:56:43,516 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 01:56:43,517 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59508 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 01:56:43,517 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1912): 4c6892182a9cd7f5a6bb4c2432955ea0/C is initiating minor compaction (all files) 2023-03-24 01:56:43,518 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4c6892182a9cd7f5a6bb4c2432955ea0/C in TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:43,518 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/020654c0a87549d3901523177615bc88, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823] into tmpdir=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp, totalSize=58.1 K 2023-03-24 01:56:43,518 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 020654c0a87549d3901523177615bc88, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679622999141 2023-03-24 01:56:43,518 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 91a4ebca2b1e4f479791eb231e77220b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679622999213 2023-03-24 01:56:43,518 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 3237788dde304070a06a8039866a5cd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679622999835 2023-03-24 01:56:43,519 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting 5bc0fa3b575140d4b94a40c2bfa12062, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1679623000499 2023-03-24 01:56:43,519 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] compactions.Compactor(207): Compacting a6a5bf5eebed47148c4b3d81ab2e7823, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679623001610 2023-03-24 01:56:43,524 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] throttle.PressureAwareThroughputController(145): 4c6892182a9cd7f5a6bb4c2432955ea0#C#compaction#619 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 01:56:43,931 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/9653db3d95f549858c8dc0685196b947 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9653db3d95f549858c8dc0685196b947 2023-03-24 01:56:43,934 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/C of 4c6892182a9cd7f5a6bb4c2432955ea0 into 9653db3d95f549858c8dc0685196b947(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 01:56:43,934 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:43,934 INFO [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/C, priority=11, startTime=1679623003094; duration=0sec 2023-03-24 01:56:43,935 DEBUG [RS:0;jenkins-hbase4:44439-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:44,322 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/34730d45572440bab7f79981764691fa as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/34730d45572440bab7f79981764691fa 2023-03-24 01:56:44,326 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4c6892182a9cd7f5a6bb4c2432955ea0/A of 4c6892182a9cd7f5a6bb4c2432955ea0 into 34730d45572440bab7f79981764691fa(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 01:56:44,326 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:44,326 INFO [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0., storeName=4c6892182a9cd7f5a6bb4c2432955ea0/A, priority=11, startTime=1679623003094; duration=1sec 2023-03-24 01:56:44,326 DEBUG [RS:0;jenkins-hbase4:44439-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 01:56:50,600 DEBUG [Thread-3894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:62686 2023-03-24 01:56:50,600 DEBUG [Thread-3894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:50,683 DEBUG [Thread-3892] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:62686 2023-03-24 01:56:50,683 DEBUG [Thread-3892] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(394): wrote 144 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4650 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4710 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4546 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4647 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(398): read 4698 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 01:56:50,683 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:56:50,683 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bafb41f to 127.0.0.1:62686 2023-03-24 01:56:50,684 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:50,684 INFO [Listener at localhost/44783] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 01:56:50,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-24 01:56:50,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:50,687 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679623010687"}]},"ts":"1679623010687"} 2023-03-24 01:56:50,688 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 01:56:50,689 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 01:56:50,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, UNASSIGN}] 2023-03-24 01:56:50,691 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, UNASSIGN 2023-03-24 01:56:50,691 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:50,692 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679623010691"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679623010691"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679623010691"}]},"ts":"1679623010691"} 2023-03-24 01:56:50,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494}] 2023-03-24 01:56:50,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:50,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4c6892182a9cd7f5a6bb4c2432955ea0, disabling compactions & flushes 2023-03-24 01:56:50,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. after waiting 0 ms 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:50,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4c6892182a9cd7f5a6bb4c2432955ea0 3/3 column families, dataSize=40.25 KB heapSize=106.08 KB 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=A 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=B 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4c6892182a9cd7f5a6bb4c2432955ea0, store=C 2023-03-24 01:56:50,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 01:56:50,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:51,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f4eb9c7fc5ba4c38a212035cff55d11e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f4eb9c7fc5ba4c38a212035cff55d11e 2023-03-24 01:56:51,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=541, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ef9449f6028341468c408756741ba15b 2023-03-24 01:56:51,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:51,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/61908f2f78fd41e38a6a3b8208d3f0a3 2023-03-24 01:56:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:52,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/b82ed37d14564146a484c75221143007 2023-03-24 01:56:52,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/A/ef9449f6028341468c408756741ba15b as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ef9449f6028341468c408756741ba15b 2023-03-24 01:56:52,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ef9449f6028341468c408756741ba15b, entries=150, sequenceid=541, filesize=25.4 K 2023-03-24 01:56:52,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/B/61908f2f78fd41e38a6a3b8208d3f0a3 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/61908f2f78fd41e38a6a3b8208d3f0a3 2023-03-24 01:56:52,082 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/61908f2f78fd41e38a6a3b8208d3f0a3, entries=150, sequenceid=541, filesize=11.9 K 2023-03-24 01:56:52,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/.tmp/C/b82ed37d14564146a484c75221143007 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b82ed37d14564146a484c75221143007 2023-03-24 01:56:52,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b82ed37d14564146a484c75221143007, entries=150, sequenceid=541, filesize=11.9 K 2023-03-24 01:56:52,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=0 B/0 for 4c6892182a9cd7f5a6bb4c2432955ea0 in 1242ms, sequenceid=541, compaction requested=false 2023-03-24 01:56:52,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3e4069b3aee4442b9adcbca452d93f30, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/9f2a3186a54c4b8eaaafc6b906bced90, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5fb272dd727d4b1e95a09460dc8c08e7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4b6addbb9e3c489aa1fbc3c4b7ebe292, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0d7214bef8ee4f3b80d5c7958fcf4ff7, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6db59438202046c5a3590ad90a8df312, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5] to archive 2023-03-24 01:56:52,088 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:52,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ab747550bef84a44886303fe3f6a6104 2023-03-24 01:56:52,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b9aaf16761a247c1a03000f8efa77e65 2023-03-24 01:56:52,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3e4069b3aee4442b9adcbca452d93f30 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3e4069b3aee4442b9adcbca452d93f30 2023-03-24 01:56:52,091 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ec8c5f421b6f4eb5bef38c278b181fb0 2023-03-24 01:56:52,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/30f75c8e4299403a9b59c60c71448159 2023-03-24 01:56:52,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/912cd5bd5e6646b6a348826371941245 2023-03-24 01:56:52,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/9f2a3186a54c4b8eaaafc6b906bced90 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/9f2a3186a54c4b8eaaafc6b906bced90 2023-03-24 01:56:52,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/671dafcd53a5498f8afb022d3f85db5e 2023-03-24 01:56:52,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1ebdd4d0b31c43628bb4fb057c428f8e 2023-03-24 01:56:52,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b4245094f87746deb43048b07593dff4 2023-03-24 01:56:52,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/b7d97d5bd2de443d81136c8fafe3ee42 2023-03-24 01:56:52,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1d1d66ac2b8f4d429cafad56c4cd40aa 2023-03-24 01:56:52,098 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5fb272dd727d4b1e95a09460dc8c08e7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5fb272dd727d4b1e95a09460dc8c08e7 2023-03-24 01:56:52,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0be12a6d5466401d968c995480be522f 2023-03-24 01:56:52,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/33a4f721acf547a7b85ca93500149b27 2023-03-24 01:56:52,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0bda28c7e52b4c52a0a9298ed1c72ace 2023-03-24 01:56:52,101 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/a3b5346dde8c4d5d87add25ce00cc10e 2023-03-24 01:56:52,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1abbab42eeec48e1baf4c4a0fcce0dc8 2023-03-24 01:56:52,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/eaefe46583b9453197db86df4aad83d3 2023-03-24 01:56:52,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4b6addbb9e3c489aa1fbc3c4b7ebe292 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4b6addbb9e3c489aa1fbc3c4b7ebe292 2023-03-24 01:56:52,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/31873ec1abf24a39be3f74a42818307c 2023-03-24 01:56:52,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/bab2df183370437399c69718e0192639 2023-03-24 01:56:52,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3beefc407096403eaebda0806c698c83 2023-03-24 01:56:52,107 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/d1c3a7cb2b1c451397be4cb0ccb2f3a2 2023-03-24 01:56:52,107 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/3cd8f632976848bf84941aa1011f1acb 2023-03-24 01:56:52,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0d7214bef8ee4f3b80d5c7958fcf4ff7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/0d7214bef8ee4f3b80d5c7958fcf4ff7 2023-03-24 01:56:52,109 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/1f614285bde244fdbb46348bb2ad9865 2023-03-24 01:56:52,110 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/91ca3ec6275042fa9c8b6665e686a15d 2023-03-24 01:56:52,111 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6d01e24ede804de487a19960ec5d3e54 2023-03-24 01:56:52,112 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6db59438202046c5a3590ad90a8df312 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/6db59438202046c5a3590ad90a8df312 2023-03-24 01:56:52,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/5ba29a3b99344f2494474fd3796f203d 2023-03-24 01:56:52,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/e12f524d5e0a4be1a505f56bc6f7d0fa 2023-03-24 01:56:52,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/8f66a3802aca492aabb928eddf72df4f 2023-03-24 01:56:52,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/4649a754206740a6a69c24dade20e0b5 2023-03-24 01:56:52,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/84493a50cb7b4d93b47049f0b84741b6, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c3de809a300c40678cb2ee5057b759c5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6c5b5d0c8e0e48d2bb4506097426169d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/86c839192d8c4b2fbe972b4ae01ee196, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/2517074e2ce34bd5babbab7e4cc94cb3, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/e43f92ef2fe745af9237bd91c2bb10ff, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/4b0d50c0c6df4f22861a183ea4a7b5cf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e] to archive 2023-03-24 01:56:52,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:52,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/964faaca01274b5895528d764a2981dd 2023-03-24 01:56:52,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/24f99e5481c9484384b2b31429b6a885 2023-03-24 01:56:52,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/84493a50cb7b4d93b47049f0b84741b6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/84493a50cb7b4d93b47049f0b84741b6 2023-03-24 01:56:52,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/30d10eae645e40bdb4c0f6de29708e15 2023-03-24 01:56:52,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/eda34b35dd794039b1ff3f7cafbf6bc3 2023-03-24 01:56:52,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c3de809a300c40678cb2ee5057b759c5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c3de809a300c40678cb2ee5057b759c5 2023-03-24 01:56:52,125 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1914d69be39344f4823aa24dedd7739e 2023-03-24 01:56:52,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6b7a15e9331548a889e85f01e82c7136 2023-03-24 01:56:52,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6618d6be9fe14ac2b59a5b1aac6269ae 2023-03-24 01:56:52,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/c0ccb0189b534bdabf1bb757533be3f4 2023-03-24 01:56:52,128 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/330c359a86124ffb9c4b239486cc2dbf 2023-03-24 01:56:52,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/8717fc4203fe46bc86c443140fb53f2c 2023-03-24 01:56:52,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6c5b5d0c8e0e48d2bb4506097426169d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6c5b5d0c8e0e48d2bb4506097426169d 2023-03-24 01:56:52,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/cac2cf3be3f9456aac3ae7893768e576 2023-03-24 01:56:52,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9b2236a1edf241969db80e51d8964ebd 2023-03-24 01:56:52,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/99214f23904a4804836707259333ab8a 2023-03-24 01:56:52,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/137457ffa4754f8f8309bb70e8b6cdbe 2023-03-24 01:56:52,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/1a0297fff5d343bfb485cbb8a5951c3b 2023-03-24 01:56:52,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/86c839192d8c4b2fbe972b4ae01ee196 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/86c839192d8c4b2fbe972b4ae01ee196 2023-03-24 01:56:52,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/3e85f814c60e4dfda547430715b70083 2023-03-24 01:56:52,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6afbe461e9da4f26a7c351bd8ec97964 2023-03-24 01:56:52,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/2517074e2ce34bd5babbab7e4cc94cb3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/2517074e2ce34bd5babbab7e4cc94cb3 2023-03-24 01:56:52,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/46ac818356864256ae2130e01dd93590 2023-03-24 01:56:52,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5dfd36efff6a4ef99dfd19223e2b49c5 2023-03-24 01:56:52,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9bc30a25b0af4bedb46b3030ba039260 2023-03-24 01:56:52,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/e43f92ef2fe745af9237bd91c2bb10ff to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/e43f92ef2fe745af9237bd91c2bb10ff 2023-03-24 01:56:52,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/356cf0377ad44cd9b947c3cf2ac3e048 2023-03-24 01:56:52,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/ad6259e61cba4202a09cddc3d4ed172e 2023-03-24 01:56:52,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/6759ed596c3743ca8c8dd05929bbcf00 2023-03-24 01:56:52,143 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/4b0d50c0c6df4f22861a183ea4a7b5cf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/4b0d50c0c6df4f22861a183ea4a7b5cf 2023-03-24 01:56:52,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/835d56caad354f7883c8171fc969ab5e 2023-03-24 01:56:52,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/5154507cb7774b5eb4fe59126bda68fa 2023-03-24 01:56:52,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/9ccb22ba3e0f4025806e8eda99177210 2023-03-24 01:56:52,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/766a32bfe14044dc96fc114977ce1e58 2023-03-24 01:56:52,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/04a2f2ef40734f5c81ec3a710f9f404e 2023-03-24 01:56:52,149 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f6706e29504847d9bf26525ca829927d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5d04ff6ac26d447f8eab4152de186e67, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/7e94dbcd894943a7ae84de0e5e622311, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8b91b4aaac754922ac367342427c5c74, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/414750f193bb46c2b3f5f7930f281c21, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/85c6cf5c92c54013af069b45535c0ba1, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/020654c0a87549d3901523177615bc88, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823] to archive 2023-03-24 01:56:52,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 01:56:52,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e6cd4cc03a084bdcbefbcfaaefa10baf 2023-03-24 01:56:52,152 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5c0e2198c8b34a5e922fe74935e6189f 2023-03-24 01:56:52,153 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f6706e29504847d9bf26525ca829927d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f6706e29504847d9bf26525ca829927d 2023-03-24 01:56:52,154 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/22b3fdea79a3489187c2d004a797a445 2023-03-24 01:56:52,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/99cf7e837a2e4297b42707eab38a1d08 2023-03-24 01:56:52,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5d04ff6ac26d447f8eab4152de186e67 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5d04ff6ac26d447f8eab4152de186e67 2023-03-24 01:56:52,156 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/72250d8a15fa4751ae95c9535affa6ac 2023-03-24 01:56:52,157 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/801f0bd4b018436a8c687c6cc277b7a5 2023-03-24 01:56:52,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2d137cb35e3e4c08a2cee45a60443ef8 2023-03-24 01:56:52,159 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/1bf6d39f63964bdf8e2880cb06744c10 2023-03-24 01:56:52,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/f5af9df027f049dbbdb0fd82469b9577 2023-03-24 01:56:52,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b115050252c34cbf8796bb121a1b99da 2023-03-24 01:56:52,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/7e94dbcd894943a7ae84de0e5e622311 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/7e94dbcd894943a7ae84de0e5e622311 2023-03-24 01:56:52,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/46fcd7c75b434ed1abc7e202e7c42a03 2023-03-24 01:56:52,163 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/36178ea8adcf427e984ec569313574bb 2023-03-24 01:56:52,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/2194008d94bb46e684e759e63f402043 2023-03-24 01:56:52,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9d86f57d0d794ba2acb8d79dc8cc8a3f 2023-03-24 01:56:52,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/cc64fb85d9db466d9363d774dc541d9c 2023-03-24 01:56:52,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8b91b4aaac754922ac367342427c5c74 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8b91b4aaac754922ac367342427c5c74 2023-03-24 01:56:52,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a4703d66cb004266a71896e7a1114cea 2023-03-24 01:56:52,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/8a9df47d7f62404597896c97d7ff4cac 2023-03-24 01:56:52,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/414750f193bb46c2b3f5f7930f281c21 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/414750f193bb46c2b3f5f7930f281c21 2023-03-24 01:56:52,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/05cc93e88756441ba164820227cb68ef 2023-03-24 01:56:52,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/665c83a0ab6340059ec1b89d645f1eed 2023-03-24 01:56:52,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/e74acc500b88430e9c604f88ba824adc 2023-03-24 01:56:52,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/85c6cf5c92c54013af069b45535c0ba1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/85c6cf5c92c54013af069b45535c0ba1 2023-03-24 01:56:52,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/86407e84243345d8a5d6ee0e219b550d 2023-03-24 01:56:52,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/70b9fb8ff9b341b2925a4736e86ac9db 2023-03-24 01:56:52,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/79f08acee97a4b4ab25e34d0ac2ad1db 2023-03-24 01:56:52,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/020654c0a87549d3901523177615bc88 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/020654c0a87549d3901523177615bc88 2023-03-24 01:56:52,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a7160495270843bc86eb4c378f86cbf4 2023-03-24 01:56:52,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/91a4ebca2b1e4f479791eb231e77220b 2023-03-24 01:56:52,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/3237788dde304070a06a8039866a5cd5 2023-03-24 01:56:52,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/5bc0fa3b575140d4b94a40c2bfa12062 2023-03-24 01:56:52,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/a6a5bf5eebed47148c4b3d81ab2e7823 2023-03-24 01:56:52,182 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits/544.seqid, newMaxSeqId=544, maxSeqId=4 2023-03-24 01:56:52,183 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0. 2023-03-24 01:56:52,183 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4c6892182a9cd7f5a6bb4c2432955ea0: 2023-03-24 01:56:52,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:52,184 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=4c6892182a9cd7f5a6bb4c2432955ea0, regionState=CLOSED 2023-03-24 01:56:52,185 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679623012184"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679623012184"}]},"ts":"1679623012184"} 2023-03-24 01:56:52,186 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-24 01:56:52,186 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 4c6892182a9cd7f5a6bb4c2432955ea0, server=jenkins-hbase4.apache.org,44439,1679622817494 in 1.4930 sec 2023-03-24 01:56:52,187 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-24 01:56:52,187 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4c6892182a9cd7f5a6bb4c2432955ea0, UNASSIGN in 1.4960 sec 2023-03-24 01:56:52,188 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679623012188"}]},"ts":"1679623012188"} 2023-03-24 01:56:52,188 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 01:56:52,190 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 01:56:52,191 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5050 sec 2023-03-24 01:56:52,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 01:56:52,792 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-24 01:56:52,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-24 01:56:52,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,795 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,795 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 01:56:52,800 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:52,802 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C, FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits] 2023-03-24 01:56:52,804 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/34730d45572440bab7f79981764691fa to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/34730d45572440bab7f79981764691fa 2023-03-24 01:56:52,805 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ef9449f6028341468c408756741ba15b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/A/ef9449f6028341468c408756741ba15b 2023-03-24 01:56:52,808 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/236bdf41542e4606a4da0c511ffb088b to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/236bdf41542e4606a4da0c511ffb088b 2023-03-24 01:56:52,809 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/61908f2f78fd41e38a6a3b8208d3f0a3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/B/61908f2f78fd41e38a6a3b8208d3f0a3 2023-03-24 01:56:52,810 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9653db3d95f549858c8dc0685196b947 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/9653db3d95f549858c8dc0685196b947 2023-03-24 01:56:52,811 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b82ed37d14564146a484c75221143007 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/C/b82ed37d14564146a484c75221143007 2023-03-24 01:56:52,813 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits/544.seqid to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0/recovered.edits/544.seqid 2023-03-24 01:56:52,814 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/.tmp/data/default/TestAcidGuarantees/4c6892182a9cd7f5a6bb4c2432955ea0 2023-03-24 01:56:52,814 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 01:56:52,814 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:56:52,815 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 01:56:52,817 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032410370d2653554df8ba2289c269ea3ad1 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032410370d2653554df8ba2289c269ea3ad1 2023-03-24 01:56:52,818 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241794f79055ac46a287d5eaee4f6ee368 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241794f79055ac46a287d5eaee4f6ee368 2023-03-24 01:56:52,818 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032417ace63ed67e4e6fa216b642f8e0f428 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032417ace63ed67e4e6fa216b642f8e0f428 2023-03-24 01:56:52,819 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032425aed75d60ca4e88bd43a11d84948cc5 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032425aed75d60ca4e88bd43a11d84948cc5 2023-03-24 01:56:52,820 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324322936ff55594881978bc84ad59f3d82 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324322936ff55594881978bc84ad59f3d82 2023-03-24 01:56:52,821 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324435ea0434d6644e0b89c321af68f8c74 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324435ea0434d6644e0b89c321af68f8c74 2023-03-24 01:56:52,822 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244847db0673b24fedb122d4c718697044 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244847db0673b24fedb122d4c718697044 2023-03-24 01:56:52,822 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032449e92aaa72f245099ac3277ed64e0ad4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032449e92aaa72f245099ac3277ed64e0ad4 2023-03-24 01:56:52,823 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324524e1c4d9d8346c8a5e24d4c4dc5d681 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324524e1c4d9d8346c8a5e24d4c4dc5d681 2023-03-24 01:56:52,824 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461dc5fd142b4493ba49774fe80f8669d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461dc5fd142b4493ba49774fe80f8669d 2023-03-24 01:56:52,825 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461f18810f2aa496ab897891fa09202f9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032461f18810f2aa496ab897891fa09202f9 2023-03-24 01:56:52,826 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324662276e9c81f42d181fc7430b3d45c1a to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324662276e9c81f42d181fc7430b3d45c1a 2023-03-24 01:56:52,826 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032476f0f590f0fc49f3ac5d1d1f5268eae9 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032476f0f590f0fc49f3ac5d1d1f5268eae9 2023-03-24 01:56:52,827 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032479906b097e504675b815523f0b5de022 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032479906b097e504675b815523f0b5de022 2023-03-24 01:56:52,828 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324929b1dc52de64de1ba7a846640078185 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324929b1dc52de64de1ba7a846640078185 2023-03-24 01:56:52,829 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032494b4c156525c4f2e9665f7501ab353d3 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032494b4c156525c4f2e9665f7501ab353d3 2023-03-24 01:56:52,830 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032496a758f2677946ab85479c579cc1ecde to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032496a758f2677946ab85479c579cc1ecde 2023-03-24 01:56:52,831 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c546bc760cc4585a7e740e0db5992a6 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c546bc760cc4585a7e740e0db5992a6 2023-03-24 01:56:52,832 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac3a5eec80204c56822f12e5ed0745cd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ac3a5eec80204c56822f12e5ed0745cd 2023-03-24 01:56:52,832 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b6b970c258df48d585e8b0b3c458b12e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b6b970c258df48d585e8b0b3c458b12e 2023-03-24 01:56:52,833 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bcf602ba6dec4b37b57158110b96a09d to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324bcf602ba6dec4b37b57158110b96a09d 2023-03-24 01:56:52,834 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d4338b00d051490c80c74a0fcd3b3d91 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d4338b00d051490c80c74a0fcd3b3d91 2023-03-24 01:56:52,835 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d452566ca94a440eadd739cfb20a8774 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d452566ca94a440eadd739cfb20a8774 2023-03-24 01:56:52,836 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d8d55eaf09f44b3588951215e065dca4 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d8d55eaf09f44b3588951215e065dca4 2023-03-24 01:56:52,837 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dcdf39f2c6f0430e84e51206fcf05aff to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324dcdf39f2c6f0430e84e51206fcf05aff 2023-03-24 01:56:52,838 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ed88493264674154ad38a2921e92b7fd to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ed88493264674154ad38a2921e92b7fd 2023-03-24 01:56:52,838 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f4eb9c7fc5ba4c38a212035cff55d11e to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f4eb9c7fc5ba4c38a212035cff55d11e 2023-03-24 01:56:52,839 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7da68ff4a204cef92f6deedc9cae1c7 to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f7da68ff4a204cef92f6deedc9cae1c7 2023-03-24 01:56:52,840 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ffb3f184558e4444a180339ec0251e4c to hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324ffb3f184558e4444a180339ec0251e4c 2023-03-24 01:56:52,841 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 01:56:52,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,843 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 01:56:52,845 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 01:56:52,846 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,846 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 01:56:52,846 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679623012846"}]},"ts":"9223372036854775807"} 2023-03-24 01:56:52,847 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 01:56:52,847 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4c6892182a9cd7f5a6bb4c2432955ea0, NAME => 'TestAcidGuarantees,,1679622979202.4c6892182a9cd7f5a6bb4c2432955ea0.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 01:56:52,847 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 01:56:52,847 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679623012847"}]},"ts":"9223372036854775807"} 2023-03-24 01:56:52,848 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 01:56:52,850 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 01:56:52,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 57 msec 2023-03-24 01:56:52,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35345] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 01:56:52,896 INFO [Listener at localhost/44783] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-24 01:56:52,908 INFO [Listener at localhost/44783] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 229), OpenFileDescriptor=543 (was 588), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 446), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=5885 (was 5769) - AvailableMemoryMB LEAK? - 2023-03-24 01:56:52,908 INFO [Listener at localhost/44783] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-24 01:56:52,909 INFO [Listener at localhost/44783] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 01:56:52,909 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:62686 2023-03-24 01:56:52,909 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:52,909 DEBUG [Listener at localhost/44783] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-24 01:56:52,909 DEBUG [Listener at localhost/44783] util.JVMClusterUtil(257): Found active master hash=47538945, stopped=false 2023-03-24 01:56:52,909 INFO [Listener at localhost/44783] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:56:52,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 01:56:52,910 INFO [Listener at localhost/44783] procedure2.ProcedureExecutor(629): Stopping 2023-03-24 01:56:52,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 01:56:52,910 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:56:52,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 01:56:52,911 DEBUG [Listener at localhost/44783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26fc59c7 to 127.0.0.1:62686 2023-03-24 01:56:52,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 01:56:52,911 DEBUG [Listener at localhost/44783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:52,911 INFO [Listener at localhost/44783] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44439,1679622817494' ***** 2023-03-24 01:56:52,911 INFO [Listener at localhost/44783] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-24 01:56:52,911 INFO [RS:0;jenkins-hbase4:44439] regionserver.HeapMemoryManager(220): Stopping 2023-03-24 01:56:52,912 INFO [RS:0;jenkins-hbase4:44439] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-24 01:56:52,912 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-24 01:56:52,912 INFO [RS:0;jenkins-hbase4:44439] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-24 01:56:52,912 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(3304): Received CLOSE for 341827374aceecc4b2088bebf463e093 2023-03-24 01:56:52,913 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:52,913 DEBUG [RS:0;jenkins-hbase4:44439] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64a6faa1 to 127.0.0.1:62686 2023-03-24 01:56:52,913 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 341827374aceecc4b2088bebf463e093, disabling compactions & flushes 2023-03-24 01:56:52,913 DEBUG [RS:0;jenkins-hbase4:44439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:52,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:56:52,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:56:52,914 INFO [RS:0;jenkins-hbase4:44439] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-24 01:56:52,914 INFO [RS:0;jenkins-hbase4:44439] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-24 01:56:52,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. after waiting 0 ms 2023-03-24 01:56:52,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:56:52,914 INFO [RS:0;jenkins-hbase4:44439] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-24 01:56:52,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 341827374aceecc4b2088bebf463e093 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-24 01:56:52,914 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-24 01:56:52,914 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-24 01:56:52,915 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1479): Online Regions={341827374aceecc4b2088bebf463e093=hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093., 1588230740=hbase:meta,,1.1588230740} 2023-03-24 01:56:52,915 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 01:56:52,915 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 01:56:52,915 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 01:56:52,915 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 01:56:52,915 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 01:56:52,915 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-24 01:56:52,916 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1505): Waiting on 1588230740, 341827374aceecc4b2088bebf463e093 2023-03-24 01:56:52,925 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/info/bc8cab4d33404eeda5fc2d39e52ae1ca 2023-03-24 01:56:52,930 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc8cab4d33404eeda5fc2d39e52ae1ca 2023-03-24 01:56:52,935 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/rep_barrier/b7722f42921949ca8496071fcec8c703 2023-03-24 01:56:52,938 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b7722f42921949ca8496071fcec8c703 2023-03-24 01:56:53,009 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-24 01:56:53,117 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1505): Waiting on 1588230740, 341827374aceecc4b2088bebf463e093 2023-03-24 01:56:53,317 DEBUG [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1505): Waiting on 1588230740, 341827374aceecc4b2088bebf463e093 2023-03-24 01:56:53,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/.tmp/info/7fb8770fa3de4ce291f59b00fda8a059 2023-03-24 01:56:53,325 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/.tmp/info/7fb8770fa3de4ce291f59b00fda8a059 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/info/7fb8770fa3de4ce291f59b00fda8a059 2023-03-24 01:56:53,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/info/7fb8770fa3de4ce291f59b00fda8a059, entries=2, sequenceid=6, filesize=4.8 K 2023-03-24 01:56:53,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 341827374aceecc4b2088bebf463e093 in 415ms, sequenceid=6, compaction requested=false 2023-03-24 01:56:53,332 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/namespace/341827374aceecc4b2088bebf463e093/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-24 01:56:53,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:56:53,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 341827374aceecc4b2088bebf463e093: 2023-03-24 01:56:53,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679622820577.341827374aceecc4b2088bebf463e093. 2023-03-24 01:56:53,344 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/table/052adbdd32b64d6ab111dbaf9bc3e890 2023-03-24 01:56:53,347 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 052adbdd32b64d6ab111dbaf9bc3e890 2023-03-24 01:56:53,347 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/info/bc8cab4d33404eeda5fc2d39e52ae1ca as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/info/bc8cab4d33404eeda5fc2d39e52ae1ca 2023-03-24 01:56:53,349 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc8cab4d33404eeda5fc2d39e52ae1ca 2023-03-24 01:56:53,350 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/info/bc8cab4d33404eeda5fc2d39e52ae1ca, entries=22, sequenceid=93, filesize=7.1 K 2023-03-24 01:56:53,351 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/rep_barrier/b7722f42921949ca8496071fcec8c703 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/rep_barrier/b7722f42921949ca8496071fcec8c703 2023-03-24 01:56:53,353 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b7722f42921949ca8496071fcec8c703 2023-03-24 01:56:53,353 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/rep_barrier/b7722f42921949ca8496071fcec8c703, entries=6, sequenceid=93, filesize=5.5 K 2023-03-24 01:56:53,354 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/.tmp/table/052adbdd32b64d6ab111dbaf9bc3e890 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/table/052adbdd32b64d6ab111dbaf9bc3e890 2023-03-24 01:56:53,356 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 052adbdd32b64d6ab111dbaf9bc3e890 2023-03-24 01:56:53,356 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/table/052adbdd32b64d6ab111dbaf9bc3e890, entries=9, sequenceid=93, filesize=5.5 K 2023-03-24 01:56:53,357 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 442ms, sequenceid=93, compaction requested=false 2023-03-24 01:56:53,360 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-24 01:56:53,361 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 01:56:53,362 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 01:56:53,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 01:56:53,362 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-24 01:56:53,517 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44439,1679622817494; all regions closed. 2023-03-24 01:56:53,524 DEBUG [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/oldWALs 2023-03-24 01:56:53,525 INFO [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44439%2C1679622817494.meta:.meta(num 1679622820316) 2023-03-24 01:56:53,528 DEBUG [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/oldWALs 2023-03-24 01:56:53,528 INFO [RS:0;jenkins-hbase4:44439] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44439%2C1679622817494:(num 1679622819714) 2023-03-24 01:56:53,528 DEBUG [RS:0;jenkins-hbase4:44439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:53,529 INFO [RS:0;jenkins-hbase4:44439] regionserver.LeaseManager(133): Closed leases 2023-03-24 01:56:53,529 INFO [RS:0;jenkins-hbase4:44439] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-24 01:56:53,529 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 01:56:53,530 INFO [RS:0;jenkins-hbase4:44439] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44439 2023-03-24 01:56:53,535 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 01:56:53,535 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44439,1679622817494 2023-03-24 01:56:53,535 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f92bbc1 rejected from java.util.concurrent.ThreadPoolExecutor@4bad26eb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2490] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:53,536 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 01:56:53,536 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44439,1679622817494] 2023-03-24 01:56:53,536 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@657d0c21 rejected from java.util.concurrent.ThreadPoolExecutor@4bad26eb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2490] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:53,537 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44439,1679622817494; numProcessing=1 2023-03-24 01:56:53,540 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44439,1679622817494 already deleted, retry=false 2023-03-24 01:56:53,540 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44439,1679622817494 expired; onlineServers=0 2023-03-24 01:56:53,540 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35345,1679622816283' ***** 2023-03-24 01:56:53,540 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-24 01:56:53,540 DEBUG [M:0;jenkins-hbase4:35345] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43e8693a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-24 01:56:53,540 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35345,1679622816283 2023-03-24 01:56:53,540 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35345,1679622816283; all regions closed. 2023-03-24 01:56:53,540 DEBUG [M:0;jenkins-hbase4:35345] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 01:56:53,540 DEBUG [M:0;jenkins-hbase4:35345] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-24 01:56:53,540 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-24 01:56:53,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679622819439] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679622819439,5,FailOnTimeoutGroup] 2023-03-24 01:56:53,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679622819439] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679622819439,5,FailOnTimeoutGroup] 2023-03-24 01:56:53,541 DEBUG [M:0;jenkins-hbase4:35345] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-24 01:56:53,541 INFO [M:0;jenkins-hbase4:35345] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-24 01:56:53,541 INFO [M:0;jenkins-hbase4:35345] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-24 01:56:53,541 INFO [M:0;jenkins-hbase4:35345] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-24 01:56:53,541 DEBUG [M:0;jenkins-hbase4:35345] master.HMaster(1512): Stopping service threads 2023-03-24 01:56:53,541 INFO [M:0;jenkins-hbase4:35345] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-24 01:56:53,542 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-24 01:56:53,542 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 01:56:53,542 ERROR [M:0;jenkins-hbase4:35345] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (334046545) connection to localhost/127.0.0.1:39055 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-24 01:56:53,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 01:56:53,542 INFO [M:0;jenkins-hbase4:35345] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-24 01:56:53,542 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-24 01:56:53,543 DEBUG [M:0;jenkins-hbase4:35345] zookeeper.ZKUtil(398): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-24 01:56:53,543 WARN [M:0;jenkins-hbase4:35345] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-24 01:56:53,543 INFO [M:0;jenkins-hbase4:35345] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-24 01:56:53,543 INFO [M:0;jenkins-hbase4:35345] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-24 01:56:53,544 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 01:56:53,544 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:56:53,544 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:56:53,544 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 01:56:53,544 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:56:53,544 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.84 KB heapSize=393.66 KB 2023-03-24 01:56:53,638 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 01:56:53,638 INFO [RS:0;jenkins-hbase4:44439] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44439,1679622817494; zookeeper connection closed. 2023-03-24 01:56:53,638 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d531c84 rejected from java.util.concurrent.ThreadPoolExecutor@4bad26eb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2490] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:53,639 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): regionserver:44439-0x101143a28860001, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 01:56:53,639 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1acfd8d4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1acfd8d4 2023-03-24 01:56:53,639 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63115b40 rejected from java.util.concurrent.ThreadPoolExecutor@4bad26eb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2490] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:53,640 INFO [Listener at localhost/44783] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-24 01:56:53,960 INFO [M:0;jenkins-hbase4:35345] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.84 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d04d6785454e4e27a5e3d785872720f4 2023-03-24 01:56:53,962 INFO [M:0;jenkins-hbase4:35345] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d04d6785454e4e27a5e3d785872720f4 2023-03-24 01:56:53,963 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d04d6785454e4e27a5e3d785872720f4 as hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d04d6785454e4e27a5e3d785872720f4 2023-03-24 01:56:53,966 INFO [M:0;jenkins-hbase4:35345] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d04d6785454e4e27a5e3d785872720f4 2023-03-24 01:56:53,966 INFO [M:0;jenkins-hbase4:35345] regionserver.HStore(1080): Added hdfs://localhost:39055/user/jenkins/test-data/a7496154-0f71-44db-dda8-aa64a59f5f97/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d04d6785454e4e27a5e3d785872720f4, entries=65, sequenceid=578, filesize=24.7 K 2023-03-24 01:56:53,966 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegion(2948): Finished flush of dataSize ~344.84 KB/353114, heapSize ~393.65 KB/403096, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 422ms, sequenceid=578, compaction requested=false 2023-03-24 01:56:53,967 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 01:56:53,967 DEBUG [M:0;jenkins-hbase4:35345] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 01:56:53,970 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 01:56:53,970 INFO [M:0;jenkins-hbase4:35345] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-24 01:56:53,970 INFO [M:0;jenkins-hbase4:35345] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35345 2023-03-24 01:56:53,972 DEBUG [M:0;jenkins-hbase4:35345] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,35345,1679622816283 already deleted, retry=false 2023-03-24 01:56:54,073 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 01:56:54,073 INFO [M:0;jenkins-hbase4:35345] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35345,1679622816283; zookeeper connection closed. 2023-03-24 01:56:54,073 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3abccca7 rejected from java.util.concurrent.ThreadPoolExecutor@a6217fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4154] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:54,074 DEBUG [Listener at localhost/44783-EventThread] zookeeper.ZKWatcher(600): master:35345-0x101143a28860000, quorum=127.0.0.1:62686, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 01:56:54,074 ERROR [Listener at localhost/44783-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@57a45577 rejected from java.util.concurrent.ThreadPoolExecutor@a6217fe[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4154] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 01:56:54,075 WARN [Listener at localhost/44783] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-24 01:56:54,079 INFO [Listener at localhost/44783] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 01:56:54,186 WARN [BP-871886-172.31.14.131-1679622812935 heartbeating to localhost/127.0.0.1:39055] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-24 01:56:54,186 WARN [BP-871886-172.31.14.131-1679622812935 heartbeating to localhost/127.0.0.1:39055] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-871886-172.31.14.131-1679622812935 (Datanode Uuid 72b7d64b-790b-4d54-8523-2c6568c41f1c) service to localhost/127.0.0.1:39055 2023-03-24 01:56:54,189 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1/dfs/data/data1/current/BP-871886-172.31.14.131-1679622812935] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 01:56:54,190 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/49cf60cb-4838-0b6d-7160-9c777bb9bf77/cluster_aaebb825-996d-ea19-3cb3-302f69415bd1/dfs/data/data2/current/BP-871886-172.31.14.131-1679622812935] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 01:56:54,226 INFO [Listener at localhost/44783] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 01:56:54,395 INFO [Listener at localhost/44783] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-24 01:56:54,579 INFO [Listener at localhost/44783] hbase.HBaseTestingUtility(1293): Minicluster is down